High Resolution and High Sensitivity Three-Dimensional (3D) Cursor Maneuvering Device

ABSTRACT

A three-dimensional (3D) maneuvering device generates 3D data by irradiating a two-dimensional (2D) reference surface with light of variable frequency. The 3D maneuvering device has high sensitivity, high resolution, and immunity from the noise caused by human vibrations. A rotational motion vector is derived by comparing the relative motion vectors derived from a pattern recognition process of surface features on the same reference plane with the positional data derived using the color index data. Thus, a single gestural movement of the operator&#39;s hand provides both the translational and rotational motion data simultaneously.

This application is a Continuation application of U.S. application Ser.No. 14/294,369, filed on Jun. 3, 2014, which claims the benefit of U.S.Provisional Application Ser. No. 61/830,834, filed on Jun. 4, 2013 andis herein incorporated by reference in its entirety.

RELATED APPLICATIONS

This application is related to docket number NU11-007, Ser. No.14/056,140 filed on Oct. 17, 2013, and herein incorporated by referencein its entirety; this application is also related to docket numberNU11-006, Ser. No. 13/834,085, filed on Mar. 15, 2013 and hereinincorporated by reference in its entirety; this application is alsorelated to docket number NU11-002, Ser. No. 13/482,086, filed on May 29,2012 and herein incorporated by reference in its entirety.

BACKGROUND 1. Technical Field

The present disclosure relates to a maneuvering system to be used forcontrolling or facilitating three-dimensional motion of a real orgraphically rendered object on a computer, on an electronic display oron other electronic or electro-mechanical devices.

2. Description

The conventional optical mouse, as illustrated schematically in FIG. 1,is a two-dimensional (2D) cursor maneuvering device. During itsoperation, the user moves the mouse (101) on a two-dimensional (2D)planar reference surface, such as a mouse pad or a desktop surface(104). By comparing a series of images of the surface captured by themouse ((102) is an image sensor, (103) is a light source) as it movesalong the reference plane, the mouse sends relative motion vectors to aremote computer or other graphical displaying device, whose displayformat is also a two dimensional one (105), i.e., the motion vectorappears with components (Δx, Δy) on a screen. FIG. 1 schematically showsthe generic structure of such a conventional art.

In recent years, three-dimensional (3D) displaying devices have becomeincreasingly popular. They can be used by a computer, a 3D printer, asurgical simulator or a video gaming device. Despite the appeal of itsclaimed functionality, a state of the art 3D displaying device suffersfrom a serious shortcoming, it cannot interact directly with theconventional optical mouse, which is a 2D device. As a result, manyprofessional graphics programs (e.g. AUTOCAD®, Medical Images, etc.)face difficulties when used in a 3D application environment that relieson a conventional 2D optical mouse. For example, positional accuracy islost and, adding to the difficulties, a significant amount of CPU poweris consumed by the matrix transformation process for the 2D mouse datathat is sent to the display. Finally, and perhaps even more importantly,the 2D mouse cannot provide angular displacement data for the graphicson the screen.

When a conventional 2D optical mouse interfaces with a graphicalrendering device that is 3D (or “2.5D”, by which is hereinafter meant 2Dwith a separate rotational axis capability), aside from the lack of theangular displacement data, there is an additional problem associatedwith the depth inherent in a 3D scene. FIG. 2 shows schematically howthe conventional optical mouse falls short in representing depth, usinga sketch of an airplane to illustrate this problem.

The skeleton of the airplane (204) is sketched by a 2.5D displayingdevice. When the operator moves the arrow (201) using a 2D cursormaneuvering device (i.e. mouse (209)) from one position to another(represented by arrow (204)), the arrow cursor may access the frame bars(202) and (203) along the path of its movement. Since what FIG. 2 showsis a 2.5D sketching system (so, the scene has depth), all the objectsdisplayed in FIG. 2 (e.g. bars, propellers, rudders, wings, etc.) havetheir respective depth values. Unfortunately, the conventional 2Doptical mouse (205) does not know all this. When a conventional 2D mouse(205) moves on its reference plane, it only senses the relativedisplacement data along X and Y coordinates; so, there is no way for theconventional mouse to ascertain or represent a depth.

The retina is a 2D organ. Using a 2D organ to analyze a 3D environmentis like superimposing a 3D coordinate system onto the image framecaptured by the conventional optical mouse. The human brain canrecognize the location of an object in a 3D coordinate system bysuperposing a 2.5D coordinate system onto the image frame he/shecaptures from the retina. But within the image sensor of a conventionaloptical mouse, there is no such 2.5D coordinate system that can be usedto inform the 3D “brain” of the display device. The prior arts revealseveral approaches to the problem of providing 3D motion to a graphicalrendering to be discussed herein. U.S. Published Patent Application2013/0002555 (Geoffrey Lee) is similar to embodiment 3 of the presentinvention, but without the 3D aspect. U.S. Pat. No. 6,720,949 (Pryor etal) discloses tracking of fingertips or other targeted tool by a camera.U.S. Pat. No. 8,291,322 (Klappert et al) teaches an input device havingan accelerometer for perspective drawings. U.S. Patent Application2008/0036773 (Bae) describes a 3D drawing system and curve editingtechniques. None of the teachings found above provide the same solutionto the problem as will be described herein.

SUMMARY

It is a first object of this disclosure to provide a 3D maneuveringsystem (alternatively referred to as a cursor maneuvering system, anavigational device or navigator and, in particular embodiments,referred to as an optical mouse or a touch-screen device) which is ableto provide 3D positional and motional data (i.e. in (x, y, z) format)without consuming excess microprocessor power and memory bandwidth.

It is a second object of this disclosure to provide such a system thatcan be used to move graphically rendered and displayed objects on adisplay device such as a computer screen or, with equal facility, tocontrol motion of an electronic or electromechanical device such as a 3Dprinter or a physical model.

It is a third object of the disclosure to provide such a system that canbe used for purposes requiring high accuracy.

It is a fourth object of this disclosure that the desired system beimmune to human body vibrations.

It is a fifth object of this disclosure that the system have anergonomic design that is “user-friendly,” and can be used comfortablyover long periods of time.

It is a sixth object of the present disclosure to provide such a systemthat supports hand movements that are linearly aligned to the movementof the cursor in the displaying device.

It is a seventh object of the present disclosure to provide such asystem that easily supports at least one vanishing point when creatingand maneuvering graphical renderings.

It is an eighth object of the present disclosure to provide such asystem that supports an easy familiarization of the correlation betweenan operator's hand motions and the resulting effects visualized on therendering device.

It is a ninth object of the present disclosure to provide such a systemthat enables artistic creativity by providing a transparency between themotions of the device and the corresponding effects of an object imageon a screen.

It is a tenth object of the present disclosure to provide such a systemthat minimizes human error.

It is a eleventh object of the present disclosure to provide such asystem that generates location and motion data using color analysis of atinted reference plane or the analysis of other properties ofelectromagnetic radiation, such as polarization or phase, that isreflected from such a reference plane or emitted from such a plane.

It is an twelfth object of the present disclosure to provide such asystem that generates location and motion data using pattern recognitionof objects on a reference plane that may or may not be tinted.

It is a thirteenth object of the present disclosure to provide such asystem wherein additional dimensions mapped from the 2D referencesurface can optionally be used to turn on or off certain functions ofthe graphic display device.

The present disclosure that will satisfy these objects is a threedimensional (3D) navigation “system”, that includes a mobilenavigational device and a special “tinted” surface on which it moves oran equivalent device with a touch-sensitive surface that replaces motionwith the effects of touch.

This system will achieve the above objects through its ability tomaneuver an object/cursor in a 3D coordinate system by measuring andextracting position and motion vectors from a two dimensional (2D)“tinted” planar reference surface (e.g. a tinted mouse pad). The systemwill also be capable of providing or facilitating control of anelectronic or electromechanical device such as a 3D printer or aphysical model. Note that although one typically associates such asystem or device with the maneuvering of a cursor or other object on acomputer screen, the present system may be equally well applied to movesome real object in space or control some electromechanical device suchas a 3D printer; thus, the use of the term “cursor” is more historicalthan descriptive and simply referring to the system simply as amaneuvering system is more accurate. Note also that the term “tinted”will hereinafter refer most generally to a treated or coated surfacethat reflects (or refracts) incident electromagnetic radiation from themobile navigational device or emits electromagnetic radiation as aresult of absorption of and re-emission of the incident radiation fromthat mobile device or other portion of the system and the tinted surfacemay, thereby, affect the intensity, color (i.e., wavelength),polarization or phase (or other physical properties that characterizeelectromagnetic radiation) of the incident radiation in its form ofemitted, refracted or reflected radiation.

Through application of this unique “tinting” method on the referencesurface, each 2D geographical point on the reference surface will alsohave a unique hue parameter (e.g. by using CIE 1931 RGB color index dataor the like to form a color gamut) and, through its hue parameter, each2D point on the reference plane can be transformed (i.e., mapped) into3D positional address data on a display screen (e.g. an x, y, and zposition), or assigned a 3D motion vector for the screen display. The 3Ddata acquired from the navigational device, purely from motions along atinted 2D surface, offers the following unprecedented advantages. First,the navigational device is able to detect the non-linear portion of thenavigator body movement (e.g. rotation, etc.) imparted to it at a speed(as measured in fps, i.e., frames per second) far higher than that of anordinary digital image capturing device. In the present navigationaldevice, the non-linear portion of device movement can be measured byanalogue circuitry which compares the hue values (converted intovoltages) of a few pixels in different colors. Thus, the performance ofthe present invention is unrestrained by frame rate. Such circuitry,embodied in the analogue data processing scheme, mimics the basicfunctionalities of bee's eyes. Human eyes rely mainly on patternrecognition to determine the whereabouts of objects in their pictureframe (e.g. pixel plane). Bees' eyes have many fewer optical receptorsbut, like human eyes, bee's photoreceptors are sensible to severalcolors (e.g. blue, yellow, etc.). In the bee's eye, however, the huesignals are not used for pattern recognition; instead, they are wiredthrough neurons to guide the bee and to allow it to navigate throughflowers without colliding with them. Judging by performance, a bee's eyeis far better than that of human being in respect of motion detection.In the present disclosure, hue is also used for motion detection,allowing the present device to perform far better than the conventionaloptical mouse.

Second, the 3D navigator delegates the data acquisition processesassociated with position (e.g. x, y, z) and motion direction (e.g. theunit vector of the plane of motion) to different features andcorresponding algorithms; this unique manner of task delegation allowsthe presently disclosed navigational device to enjoy high positionalaccuracy, which is mainly attributed to the block matching algorithms(BMAs) used by the presently disclosed device (as also taught incommonly assigned application NU11-007, fully incorporated herein byreference), and the ability to control the non-linear motions ofcursors/objects, which is mainly attributed to the hue sensitivity ofthe device. The deliberate task delegation process as is stated aboveallows the presently disclosed system to perform 3D motion detectiontasks easily, without paying much penalty in calculation load.

Third, the positional data can be cross-referenced to a coordinatesystem that is embedded in a “coloration” reference plane (e.g. tintedmouse pad). The tinted reference plane provides a color gamut containingthe hue data for the presently disclosed navigation device in acontrollable manner, because the hue data varies from each geographicpoint to another in a gradual manner. Specifically, along a specificdirection (e.g. the x-axis of the coordinate system embedded in thetinted reference plane), a parameter of hue (say CIE red color stimuli),will increase or decrease linearly along that direction. When there arethree such kinds of hue variation profiles (e.g., tri-color stimuli)imprinted onto the tinted reference plane in an independent andnon-interfering manner, the navigation device is able to acquireabsolute 3D positional data from what is effectively an embedded 3Dcoordinate system which is actually determined by the tinting conditionof the reference plane.

An ordinary computer relies on tracing targeted objects (e.g. using BMA,block matching algorithm) to determine the displacement of targetedobjects in monochromatic image frames. When the motion of the devicebody is subject to extreme motions, a conventional optical mouse mayeasily lose track of the targeted object; in this situation, thecomputer may cease operating. With the operational features of thepresent navigational device, even after an extreme movement of theoperator's hand, the device can still rapidly determine the positionaldata, which is a feature greatly desired by video game applications.Thus, the relation between the 2.5D coordinate system of the tintedreference plane and the 3D coordinate system used by the 3D graphicrendering system can be cross-checked or re-established at all times.

A bee also uses similar abilities to take a lengthy journey (e.g.several miles). Despite the fact that bees never see any objectsclearly, they can take a long distance journey without getting lost. Thefundamental capability that enables a bee to do such an amazing job lieson the hue map stored in its memory. This map is recorded together withthe sunlight illumination condition (gradient of a sun ray denotes thedirection to the point of infinity). If an engineer uses a sketch pad tosimulate the bee's navigation map, it is as though the pad has a 3Dcoordinate system and a vanishing point. In a bee's 3D navigation map,there are many vectors (hue gradient is a vector), and there is areference point of infinity, yet no object has a clear contour. In thenext generation 3D optical mouse, these features will be built into theproduct. Thus, unlike the prior art which relentlessly pursues higherand higher frame rates, or uses coherent light sources, etc., thepresent invention disclosure provides a 3D navigation map. In thefollowing paragraph, we will briefly review the functionality of aconventional optical mouse, and then illustrate the importance of havingsuch a 3D navigation map.

An image presented by a 2.5D graphical rendering system mimics a human'sneural system (i.e. the pixel frame is in fact a 2D one). The patternrecognizing capability of human eyes endows us with the capability ofextracting features from the pixel frames (i.e. the retina). Afterimposing an artificial 2.5D coordinate system onto the pixel frame, thefeature being extracted out of said pixel frame will have a 3Dpositional address. Such an image processing scheme requires manycalculations; hence, human beings cannot see things clearly when a scenechanges too rapidly. In nature, a human is a creature that is slowmoving as compared to most other animals. On the other hand, a bee'seyes are designed to cope with rapid motions in an ever changingenvironment. When a bee flits from one flower to another, the image asis seen by bee's eyes is totally changed (this may just take a fewseconds). If the bee used a human perception system, it would becomedizzy. Fortunately, bees' eyes are far better than that of human beingin this respect, they are able to detect motion at high speed withoutgoing through pattern recognition process. The hue data serves as afield of optical flow; so, to fly through flowers, a bee does notnecessarily depend on any image that has a vivid contour of objects. Ifone uses a mathematician's theory to depict the bee's neural system,he/she would understand that the bee's eyes work by measuring thegradient of light flows in the different colors. Literally speaking,bee's compound eyes are a cluster of optical flow equations. When ahuman desires to improve their current product optical mouse, that humanbeing can learn from Mother Nature. What the next generation opticalmouse really wants to be, is a hue-sensitive one. With theseunderstandings in mind, it becomes clear that the present inventionrepresents the onset of a new era of the computer graphic industry,steering toward hue detection methodology. On the other hand, the deviceengineer must acknowledge that detecting motion only by hue is not ananodyne to the conventional art, there are limitations. A bee's eyes arenot accurate in terms of locating objects; but the optical mouse used bya human being would require high positional accuracy. Thus, if a deviceengineer designs navigational circuitry based only on the anatomy of abee's eyes, the device will have a high responsiveness to non-linearmotions (i.e. be responsive to optical flow variations), but will havelow resolution and poor positional accuracy. One can envision thedrawback of the bee's eye by the following scenario. If some(potentially dangerous) object, larger than the bee, is approaching thebee slowly, the bee wouldn't be able to tell any danger is near becauseits eyes are not designed to recognize the object precisely. The 3Dnavigation device we human beings are intending to develop is one thatrequires high positional accuracy. In many applications (e.g. AUTOCADdrawing, 3D robot assisted medical surgery, etc.), positional accuracyremains a fundamental feature that cannot be sacrificed to the newrequirement on non-linear motion sensing capability. Ever since themodern optical mouse was invented about twenty years ago, the demand forpositional accuracy has been an emphasized requirement.

In order for it to acquire a 2D motion vector at high positionalaccuracy, a digital camera is mounted in a cavity underneath the mouse.Using a block matching algorithm (BMA), a series of pictures of thedesktop surface are compared in a timely manner. Such a design rule hasbeen utilized by the computer industry for a long period of time. Eventoday, the conventional optical mouse still literally controls nothingbut the translational movement of an arrow shaped cursor on a 2D plane(i.e. displaying panel or computer screen). In 3D applications, thecursor (i.e. now a 3D object) does not necessarily lie on a 2D plane.Thus, both the positional accuracy and non-linear motion controllabilityare required for performing comprehensive 3D cursor maneuvering tasks.Facing these new requirements, the conventional optical mouse isoutdated. There now emerges an imminent demand for the microelectronicindustry to develop new means to detect the high order terms of motionvector (in 3D format) while maintaining a fairly good positionalaccuracy. As was stated above, the anatomy of the bee's eyes provides uswith the clue. Their eyes recognize a few colors (i.e. hue), but theirneurons respond to the variation of hue in a much different way thanthat of human eyes. The bee uses hue primarily as a means to detectmotion, much less for pattern recognition. The merit of the bee's neuralsystem has been amply demonstrated by observation of their agile andprecise activities when they fly swiftly through a complicated 3Dcoordinate system, such as a flower-covered bush. The ecology ofcomputer industry is nothing more or less than that of Nature, in whichDarwin's rule prevails. That is, the conventional optical mouse industryis facing a critical point when the 3D graphic era must survive throughimprovement, or be extinguished. To improve the performance of anoptical mouse, it is very natural that a device engineer will desire toadd the non-linear motion controlling capability to the mouse unit. Whena device engineer designs an electrical circuit that mimics the basicfunctionality of the bee's eyes, they may discover that the circuitry ismuch like an analog circuit (e.g. differential amplifier). A swiftmotion of an object from the left to the right side of the bee may causecertain neuron pairs in bee's eyes to have the opposite signal profilesas compared to that of another swift motion from right to left (e.g. A>Bvs. A<B, where A and B are the intensities of the generated neuralsignals). The differential amplifier fabricated by modernmicroelectronic circuitry can have such a sensitivity, provided itsinput nodes are properly linked to certain hue detecting devices. Inorder to meet this goal, the optoelectronic sensors used by thepresently disclosed navigational device must be hue-sensitive ones (e.g.a color sensitive CMOS sensor, etc.), and the planar reference surfaceused by the device must be a tinted surface. Thus, when the presentlydisclosed navigational device moves over the tinted reference plane,both the positional data as well as directional data (i.e. non-linearmotion vector) will be generated concurrently (in commonly assigned Ser.No. 14/056,140, Filing Date Oct. 17, 2013, fully incorporated herein byreference, the two parameters are derived from the same “lumped” data).In most practical cases, the apparent color of the tinted referenceplane will be composed of three primary colors (more or fewer colors arealso possible, but they will be used for special situations). When thephoto detector (e.g. photo-diode) in the device is responsive to theseprimary colors, the hue data as measured by the present invention areorthogonal (i.e., the red color index data will not interfere with thatof blue colored light, and vice versa). Thus, the hue-sensitive digitalcamera embedded in the presently disclosed navigational device may“feel” that it is moving in a 3D space, just as a bee sees itssurroundings when it flies through the flowers. As a result of theseunique features, simply by moving the presently disclosed device over atinted 2D plane (e.g. a tinted mouse pad), a corresponding object/cursorin a 3D graphic rendering device may be maneuvered. Moreover thismaneuvering act concurrently enjoys high positional accuracy as well ashigh sensitivity to the non-linear motions. Thus, the presentlydisclosed device opens a new door for the electronic industry throughwhich to enter into the 3D virtual reality regime.

From a physical point of view, a 3D graphic rendering device is an imageprocessor, and an optical mouse is also an image processor (which storesdata in pixel planes). The fundamental differences between the two imageprocessing devices can be illustrated by reference to Eq's. (1) and (4).Consider a pixel, which can be considered as simply any object to bemoved by the optical mouse. The pixel, denoted simply as P, is locatedat point P in a 3D orthogonal coordinate system. When pixel P is movedby a combination of rotational and translational displacements, whoseparameters as are denoted by operators R and T in Eq. (1), it will reacha new position, denoted by point P′.

$\begin{matrix}{P^{\prime} = {{{R \cdot P} + T} = {\begin{bmatrix}X^{\prime} \\Y^{\prime} \\Z^{\prime}\end{bmatrix} = {{\begin{bmatrix}R_{11} & R_{21} & R_{13} \\R_{21} & R_{22} & R_{23} \\R_{31} & R_{32} & R_{33}\end{bmatrix} \cdot \begin{bmatrix}X \\Y \\Z\end{bmatrix}} + \begin{bmatrix}T_{X} \\T_{Y} \\T_{Z}\end{bmatrix}}}}} & (1)\end{matrix}$

As is illustrated by Eq. (1), the rotational displacement R is not asingle valued object, it is represented by a (3×3) matrix, i.e.,

$\begin{bmatrix}R_{11} & R_{12} & R_{13} \\R_{21} & R_{22} & R_{23} \\R_{31} & R_{32} & R_{33}\end{bmatrix};$

likewise, the translational displacement data T is not a singleparameter data, it is a (3×1) matrix containing three translationaldisplacement parameters, denoted as

$\begin{bmatrix}T_{X} \\T_{Y} \\T_{Z}\end{bmatrix}\quad$

in Eq. (1). As the consequence of these motions, the final position ofP′ is represented by

$\begin{bmatrix}X^{\prime} \\Y^{\prime} \\Z^{\prime}\end{bmatrix}.$

The rotation matrix R has several formats. As Eq. (2) shows, therotation matrix R can be deemed as a combination (i.e. successive matrixmultiplications) of three rotational matrixes, each of which designatesa rotational motion around a single axis (i.e., x-axis, y-axis, andz-axis). Thus, as Eq. (2) shows, the rotational displacement of a pointin a 3D coordinate system can be represented by three angulardisplacement data, i.e. ϕ, φ, θ. Substituting ϕ, φ, θ for R₁₁˜R₃₃ in Eq.(1), one can re-write the rotation matrix R, as is depicted in Eq. (2),and vice versa.

$\begin{matrix}{R = {{\begin{bmatrix}{\cos \; \varphi} & {\sin \; \varphi} & 0 \\{{- \sin}\; \varphi} & {\cos \; \varphi} & 0 \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}{\cos \; \phi} & 0 & {{- \sin}\; \phi} \\0 & 1 & 0 \\{\sin \; \phi} & 0 & {\cos \; \phi}\end{bmatrix}}\begin{bmatrix}1 & 0 & 0 \\0 & {\cos \; \theta} & {\sin \; \theta} \\0 & {{- \sin}\; \theta} & {\cos \; \theta}\end{bmatrix}}} & (2)\end{matrix}$

Alternatively, the rotational matrix R can also be depicted by Eq. (3),in which the parameters n₁, n₂, and n₃ denote the unit vector of therotational axis {right arrow over (n)} (i.e. the pivot axis), ω denotesthe rotational angle around said rotational axis n.

$\begin{matrix}{R = \begin{bmatrix}{n_{1}^{2} + {\left( {1 - n_{1}^{2}} \right)\cos \; \omega}} & \begin{matrix}{{n_{1}{n_{2}\left( {1 - {\cos \; \omega}} \right)}} -} \\{n_{3}\sin \; \omega}\end{matrix} & \begin{matrix}{{n_{1}{n_{3}\left( {1 - {\cos \; \omega}} \right)}} -} \\{n_{2}\sin \; \omega}\end{matrix} \\\begin{matrix}{{n_{1}{n_{2}\left( {1 - {\cos \; \omega}} \right)}} +} \\{n_{3}\sin \; \omega}\end{matrix} & {n_{2}^{2} + {\left( {1 - n_{2}^{2}} \right)\cos \; \omega}} & \begin{matrix}{{n_{2}{n_{3}\left( {1 - {\cos \; \omega}} \right)}} -} \\{n_{1}\sin \; \omega}\end{matrix} \\\begin{matrix}{{n_{1}{n_{3}\left( {1 - {\cos \; \omega}} \right)}} -} \\{n_{2}\sin \; \omega}\end{matrix} & \begin{matrix}{{n_{2}{n_{3}\left( {1 - {\cos \; \omega}} \right)}} -} \\{n_{1}\sin \; \omega}\end{matrix} & {n_{3}^{2} + {\left( {1 - n_{3}^{2}} \right)\cos \; \omega}}\end{bmatrix}} & (3)\end{matrix}$

In practice, depending on the situation, one may select Eq. (2) or (3),or the other formats of rotational matrix R (e.g. a quaternion, etc.)for his/her own use, and the outcome should be the same. For example,Eq. (2) requires three angular data to describe a rotational movement(ϕ, φ, θ); as for Eq. (3), only one angular displacement data isrequired (ω). The present navigational device uses Eq. (3) to derive therotational matrix R.

Physically speaking, a prior art optical mouse is a mobile image sensorthat takes successive pictures of a 2D planar reference surface (e.g.the desktop surface, a mouse pad, or the like), on which there lies aplurality of targetable microscopic objects (e.g. dust, surfaceimperfections or topological features generally). When there is arelative motion between the image sensor and the reference plane, acorresponding movement of these targetable objects takes place in theimage sensor, which is usually formed by a plurality of pixels. Themotion of these pixels can be represented by Eq. (4), in which theparameter M denotes the rotational displacement, and the parameterT_(Optical-mouse) denotes the translational displacement, respectively.

$\begin{matrix}{P_{Optical\_ mouse}^{\prime} = {{{M \cdot P_{Optical\_ mouse}} + T_{Optica\_ mouse}} = {{\begin{bmatrix}{\cos \; \vartheta_{x}} & {{- \sin}\; \vartheta_{y}} \\{\sin \; \vartheta_{x}} & {\cos \; \vartheta_{y}}\end{bmatrix} \cdot \begin{bmatrix}X \\Y\end{bmatrix}} + \begin{bmatrix}T_{X,{2D}} \\T_{Y,{2D}}\end{bmatrix}}}} & (4)\end{matrix}$

Eq. (4) is the most general mathematical representation of a 2D motionsensing camera. The typical conventional optical mouse found in today'sconsumer electronic market can only measure the translationaldisplacement data, i.e.

$\begin{bmatrix}T_{X,{2D}} \\T_{Y,{2D}}\end{bmatrix}\quad$

of Eq. (4). In the past, there were only very few inventions attemptingto measure the rotational term M of Eq. (4). Commonly assigned Ser. No.14/056,140, Filing Date Oct. 17, 2013, fully incorporated herein byreference, discloses an optical mouse that also measures the rotationaldisplacement data M of Eq. (4) by extracting it from a lumpedtranslational motion vector data. This device allows the next generationcursor maneuvering device to have a rotational functionality withoutrequiring an operator to change his/her habits of moving the mouse insome exaggerated fashion.

In a 2.5D system, the positional data originally in 2D format can beconverted to a 3D format. Thus, when the translational data androtational data are converted into 3D format, an optical mouse is ableto determine the final position P′. By imposing a 2.5D coordinate systemonto the images captured from the reference plane, the present deviceallows a computer to recognize that the data acquired from the presentlydisclosed navigational device is 3D data. In the past, the positionaldata measured (and transmitted) by the optical mouse was only 2D data,so the computer had to exercise a 2D to 3D data transformation processto generate the 3D motion vector it needed (e.g. ΔX, ΔY, ΔZ). Becausethere is no validated physical relationship between the mouse andcomputer, the above stated transformation process requires extra CPUcomputation power, and it does not intuitively correspond to the handmovement of the operator. The consequence of the above processes is thatthere is a loss of realism in the motion of the 3D object as we normallysee in today's 3D graphic rendering device, and the interaction betweenthe operator and computer is restricted.

One of the objectives of the present disclosure is to provide 3Dpositional data from a 2D planar reference surface by measuring its“tint” or other optical features of an electromagnetic wave that isreflected from or emitted by that surface, and to establish a 2.5D to 3Ddata transformation process from that measurement. If we shine threelight rays of different colors (e.g. R, G, B) onto a 2D referencesurface that is reflectively coated, then every point on the referencesurface can reflect light in a hue value that is denoted by a set oftri-stimuli data (e.g. CIE1931 RGB). After data conversion, the presentdevice is able to provide the positional data in 3D format. Hence, bymoving the present mouse (i.e., navigational device) body on a “tinted”2D reference surface (e.g. desktop surface), 3D positional data isprovided for the computer to use. Thus the hue data contained in thecolor gamut provides a method of mapping the 2D surface of the planarreference surface into the 3D space of the graphic rendering device (orother electronic or electromechanical device) in 3D format. The priorart optical mouse is a monochromatic device; therefore, it lacks thiscapability of providing motion data in 3D format. FIGS. 3A and 3Bschematically show the relationship between a 2.5D coordinate system anda 3D coordinate system.

As FIG. 3B shows, there is a geometric relationship between a 2.5Dcursor maneuvering device and a 3D coordinate system. That is, for anarbitrary point that is labeled as (x, y, z) in a 3D coordinate system(e.g., the black dot in 3A), its whereabouts in a projected 2D plane canbe denoted as (x′, y′) and the two sets of coordinate data (i.e. 2D and3D) can be converted from one to another in accordance with Eq's. (5A)and (5B), respectively.

x′=x−z cos σ  (5A)

y′=y−z sin σ  (5B)

As is shown in the exemplary illustration of FIG. 3A, the 3D object(i.e., the cube) has all of its edges intersecting at a 90 degree angle(0=90). Depending on the perspective angle chosen, the correspondingangle σ of the cube projected in FIG. 3B is a parameter chosen by the2.5D displaying device (schematically denoted as (301B), which isliterally a 2D plane from a physical point of view, with a being about45 degrees). From Equ's. (5A) and (5B), one can understand that when thetinted reference plane is illuminated by a specific lighting condition(i.e., intensity), the mathematical plane function formed by the set ofall geographical points on the reference plane is nothing more than thatof a 2.5D coordinate system. However, if there are changes in thelighting condition or the shutter speed of the image sensor, or otherfeatures of the presently disclosed device that produce a similareffect, then the mathematical function of said 2.5D plane will becorrespondingly changed. Mathematically, this is equivalent to sayingthat the 3D plane has changed its unit normal vector. Using this method,the present disclosure provides a revolutionary navigational device inthe form of a 3D optical mouse that, by a single movement of theoperator's index finger, can change the direction of the 3Dcursor/object. Furthermore, by that single movement of the devicerelative to the reference plane, the 3D cursor/object can be maneuveredby the translational displacement data and rotational displacementconcurrently. Based on Eq. (1), the above set of data meets therequirement sufficiently for maneuvering an object/cursor in a 3Dgraphic rendering system.

In the present disclosure, the rotational matrix R of Eq. (1) can becalculated using Eq. (3). According to Eq. (3), a rotational matrix Rcan be derived as a function of a rotational angle ω and a unit vector{circumflex over (n)}, where {circumflex over (n)}=(n₁, n₂, n₃). Inpractice, n₁, n₂, n₃ can be given by the computer program; or, it can beadjusted by the presently disclosed navigator (e.g. (401) of FIG. 4). Atthe moment, for easier narration, we take n₁, n₂, n₃ as the knownfactors. FIG. 6A schematically shows the phenomenon of moving thepresently disclosed navigator without changing the parameters n₁, n₂,and n₃. When n₁, n₂, and n₃ are not changed during the navigationmovement, the arrow shaped cursor (604) will only move in plane (603A)in the 3D graphic rendering device (608A). In this exemplary case, wedenote plane (603A) as an aisle. Items (606A) and (602A) are the wallson both sides of aisle (603A). Thus, regardless the motion of thenavigator (502) on the reference plane (510), cursor (604) will notcrawl onto the walls (606A) and (602A). This gives the operator a morerealistic feeling.

Inside the computer memory, there is a place that stores the 3Dcoordinate system and the addresses of those features (e.g. aisle andwalls). Specifically, as is shown in FIG. 6C, the arrow shaped cursor(604) denotes point C in said 3D coordinate system. By the same token,point C of FIG. 6C corresponds to point B on the planar referencesurface (606), as is shown in FIG. 6B. Whenever the operator moves thenavigator (607) by a motion vector (Δx*, Δy*) on the reference surface(606) (e.g. on a mouse pad), the point C in the 3D coordinate systemwill be moved by a corresponding motion vector (Δx**, Δy**), and thearrow cursor (604) will be moved in the displaying device (608A)correspondingly. Note again that, unless the unit vector {circumflexover (n)} in FIG. 6C is changed, regardless of whatever motion vector isacquired from motion relative to the reference plane (606), point C willnot be moved out of the plane a₁b₁c₁. In the graphic rendering device,such a situation implies that the 3D cursor 604 will not be moved out ofarea (603A) in FIG. 6A. This unique feature offers unprecedentedadvantages for a 3D GUI (Graphical User Interface) that requires highpositional accuracy. As is well known in the state of the art opticalmouse industry, an optical mouse can easily reach a resolution ofhundreds, or thousands dpi from a planar reference surface. But theresolution of a state of the art 3D game controller cannot easily matchthis specification. Thus, the presently disclosed navigator can serve asboth a high sensitivity and high resolution 3D cursor/object maneuveringdevice for advanced 3D applications, the premise being that it has acapability to manipulate the unit vector {circumflex over (n)}. Thisissue has been implemented by a unique feature (i.e. touch sensitivefeature (401) of FIG. 4) provided by the present invention, which willbe elaborated in the latter paragraphs as an additional embodiment. Inaddition, related application Ser. No. 14/056,140 Filing Date Oct. 17,2013, which is fully incorporated herein by reference, has disclosed amethod and a corresponding navigational device for implementing thatmethod for providing rotational displacement data by measuring therelative movement of that navigator on a 2D reference plane. In a likemanner as in commonly assigned Ser. No. 14/056,140, Filing Date Oct. 17,2013, as schematic FIG. 7B shows, the presently disclosed navigator isable to provide a rotational motion data (ω*) during the course of itstranslational movement (Δx*, Δy*); as shown in schematic FIG. 7C, thecomputer recognizes the respective motion parameters as (Δx**, Δy**) andω**. In the corresponding 3D graphic rendering device as is shown inFIG. 7A (or any device that has 3D motion features and capabilities,such as a 3D printer), cursor (704) is subjected to the translationalmotion vector (706) as well the rotational motion vector ω***. Note thatboth the translational and rotational motions of cursor (704) occur onplane (703A). Thus, there will be no out-of-plane motions for the cursor(704) because the unit vector {circumflex over (n)}=(n₁,n₂,n₃) of planea₁b₁c₁ is not changed in this case. In many 3D applications (e.g. videogames, AUTOCAD™), this feature provides unprecedented advantages for theoperator in that the 3D positional data provided by the presentnavigational device can move an object in a reference plane that is veryfamiliar to the operator. Compared to most of the present mass-markethaptic devices (e.g. EM trackers, depth camera, wii, etc.), the presentdevice is far more accurate, sensitive, and adaptable to the operator,making itself particularly useful to the professional who works in suchareas as surgery, based on a use of 3D medical images.

When the parameter n₁, n₂, or n₃ is changed, the point of intersectionbetween the plane and the three axes of the coordinate system will bechanged accordingly. As FIG. 8C shows (an exemplary case), when the unitvector changes, inside the computer memory or in the navigator'sinternal memory the plane of cursor motion will be changed from a₁b₁c₁(whose unit vector is {circumflex over (n)}) to a₁ b₁″c₁ (whose unitvector is {circumflex over (n)}″). Correspondingly, in the 3D graphicalrendering device, the plane of cursor motion has been changed to (803A).Following this plane (803A) tilting action, cursor (804) is now more“upwardly” tilted as compared to that of FIG. 8A. Thus, due to the newplane position, cursor (804) is able to reach point (806), the ceilingof the aisle, which was not the case for the cursor of FIG. 7A. By thesame token, as FIG. 9C shows, when the cursor motion plane has beenchanged to plane a₁b₁*c₁, cursor (904) will be moving along plane(903A). As a result, cursor (904) is unable to reach point (901A), theend of the aisle, whereas the cursor in FIG. 7A can.

In practice, the operator may perform all of the above described 3Dcursor maneuvers using one hand (i.e. cursor maneuvering, and planetilting), controlled by a collaboration between finger gestures (e.g.index finger), wrist motion, elbow and arm movement, etc. As FIG. 10Bshows, the operator may move the navigator (1007) by a translationalmotion vector (Tx, Ty, Tz), and a rotational motion vector ω; during thecourse of these combined motions, the operator may wiggle his/her indexfinger (1009) over a unit vector adjustment device (1008) (e.g. aLambertian touch sensitive plate to be described below as a separateembodiment). The point I (more precisely, the area) where the operator'sfinger contacts the touch sensitive device, denotes the value of n₁, n₂,and n₃ of the unit vector {circumflex over (n)} of the cursor motionplane. Hence, as FIG. 10A shows, the arrow cursor moves from (1004A),through (1004B), and finally, by the combination of the operator'sfinger and hand movement reaches (1004C). The above mentioned Lambertianplate has been disclosed in commonly assigned related application Ser.No. 13/482,086, Filing Date May 29, 2012 and Ser. No. 13/743,582, FilingDate Jan. 17, 2013, both fully incorporated herein by reference. Inpractice, there are still other ways to acquire the 3D vector (i.e. unitvector of the mouse maneuvering plane) by finger movement. For example,a “wiggling knob” (a specific control device on the navigator) can beconfigured to do the same job, but its resolution usually is lower thanthat of a Lambertian touch sensing plate. To the viewers, all of theabove stated cursor positions are changed smoothly and continually Thecursor/object's motion follows the continuous motion of the operator'shand's and finger's movement immediately and responsively, making thepresently disclosed navigational device to be a 3D cursor/objectcontrolling device of unprecedented qualities. Note that one should notequate the presently disclosed navigator with the “gesture sensingdevice” cited in many prior art references. FIGS. 11 and 12schematically show two kinds of gesture sensing devices provided by theprior art. Specifically, the device of FIG. 11 uses electromagnetic (EM)waves to detect the motion of the operator's hand (1102) (which grips anEM tracker). As for FIG. 12, a depth sensitive camera (1204) isobserving the movement of the operator's hand (1202). If one observesthe mechanistic performance of these two devices, they will realize thatbecause a human body is flexible, human body vibration is, therefore,inevitable in most situations. As the schematic models of FIGS. 11 and12 show, if we use springs (1103) and (1203) to denote the mechanism ofbody vibration, various types of vibrational modes can be imposed on theoperator's hand (1102) and (1202). Thus, the motional data measured bydetectors (1104) and (1204) are constantly generating errors (i.e. (±x,±y, ±z)). In professional applications (e.g. manipulating medicalimages, aerospace 3D parts fabrication, etc.), errors incurred by theoperator's hand vibration are not desired. FIGS. 13A and 13B show thisvibrational phenomenon. In both figures, the operators who are beingtracked by the detectors (e.g. 1104 or 1204) are in different bodypositions (e.g. FIG. 13A, sitting; FIG. 13B standing), the motionvectors generated by the two body gestures (i.e. one in FIG. 13A, theother FIG. 13B) are subjected to different kinds of vibrational errorsas a result of the different body positions. Using the presentnavigation device as shown in FIG. 5, the navigator (502) is positionedover a supporting plane (510). Hence, human body vibrations can besuppressed. This seemly simple and conventional method of devicenavigation has somehow been discarded by many 3D cursor controllingdevices. Inside the presently disclosed navigator, motion detectioncircuitry is constantly comparing the images captured at different timeintervals. A typical method used by this circuitry is the block matchingalgorithm (BMA) as discussed in Ser. No. 14/056,140, Filing Date Oct.17, 2013. When the frame rate is very high, the motion vector calculatedby the BMA method can fall within a range as small as a few pixels (e.g.a few μm), with a small amount of deviation that is possibly caused bythe non-linear portion of mouse movement.

The present invention makes use of methods previously disclosed incommonly assigned Ser. No. 14/056,140, Filing Date Oct. 17, 2013 toseparate out the non-linear portion of the present navigator's motionvector. Thus, rotational motion data can be extracted in a way thatallows the data of the rotational matrix R in Eq. (3) to have aconsistent interpretation at all times. Together with the translationalmotion data (which is already very sensitive and of high resolutionbecause of the use of state of the art image capturing techniques), thedata acquired by the presently disclosed navigator uses the imageprocessing methods of physics to produce high sensitivity and highresolution 3D cursor maneuvering capabilities.

As FIG. 14A schematically shows, the next generation cursor maneuveringdevice will have to operate on a great variety of surfaces, e.g., a sofasurface (1403), bench top (1404), or a tiled floor section (1405), etc.These surfaces have vastly different topographical properties (i.e., therespective SQUAL values vary greatly). If a cursor maneuvering devicecannot operate on at least some of these surfaces in a stable andpredictable manner, it will not satisfy those family members who maytypically situate themselves in a specific area in the living room wherea firm, stable, planar reference surface is not available. So, afundamental requirement for the next generation optical mouse, if it isto succeed in the market, is a wide adaptability to various kinds ofsurfaces. If we translate such a requirement to the terminology used bythe mouse industry, it means that the range of SQUAL values for nextgeneration optical mouse will be far wider than that of today's mouse.So, the present navigational device, which is envisioned herein as beingthat next generation optical mouse, is not necessarily a device thatpursues extremely high dpi (dots per inch) or operability on a glossysurface (some prior arts believed this is the key selling point), but itis one having wide adaptability to various kinds of surfaces. Relatedapplication Ser. No. 13/834,035, Filing Date Mar. 15, 2013 fullyincorporated herein by reference, provides this feature. Besides surfaceadaptability, there is another requirement for next generation opticalmouse, an integrated 2D and 3D cursor maneuvering functionality. As FIG.14A shows, this requirement is incurred because the same displayingdevice (1401) in the living room will be used by different applications.For example, when the exemplary flat panel displaying device (1401) isused by a family member to play 3D video games, the present inventionmust work as a 3D navigator. When the flat panel displaying device(1401) is used by another family member to read news on a web site (anews report is a 2D plane), the presently disclosed navigational deviceworks as a 2D cursor maneuvering device. In the future, swapping between2D and 3D scenes on the same display and in the same application canhappen suddenly and often unexpectedly. An exemplary case is illustratedin schematic FIG. 14C. When the operator moves an arrow shaped cursorusing the present navigational device, the position of that cursorchanges from (1426A) to (1426B) and finally to (1426C). The 3D displaydevice (its screen shown here as (1425)) is aware that the cursor is ina 3D space and the instruction menu bar (1423) is defined by the displayas being located at the side nearest the operator. Thus, neither of thethree cursor positions (1426A) to (1426C) can cross over the menu bar(1423).

A 2D mouse cursor (shown as dark arrow (1424)) would be able to accessthe instruction bar when it is close to the bar, but it cannot tell thatthere is a vanishing point for perspective (1421) in the display (1425).By integrating the above two functionalities (2D and 3D) in one system(i.e. a combined 2D/3D optical mouse), the present system of navigatorplus surface allows the operator to enjoy the convenience of havingthose two features on one product body. There is a further uniquecharacteristic of the presently disclosed navigator. On its tintedreference surface, every geographic point will be able to provide either3D positional data or 2D positional data, or both, in a correspondingway (i.e. they can be converted back and forth). Under a severecircumstance, such as the navigation device being accidentally “jolted”by an action of operator's hand (such as when an operator is playing afierce battleground video game), wherein the morphology of the imagestaken by the navigator lose their correlations, the presently disclosednavigation device may occasionally lose track of the targeted objects.In this situation, the tinted reference plane can support the presentlydisclosed navigator to quickly find out its whereabouts by measuring thehue value from the tinted reference plane.

To meet the above the requirement, the presently disclosed system uses acolor sensitive feature (i.e. comprising multiple-wavelengthilluminating system and a wavelength sensitive sensor) to trace themicroscopic objects. According to color physics, a set of color indexdata such as CIE1931 RGB can be converted to another set of data such asX, Y, Z by Eq. (6).

$\begin{matrix}{\begin{bmatrix}X \\Y \\Z\end{bmatrix} = {\begin{bmatrix}D_{{A = 1},\; {B = 0},\; {C = 0}} & D_{{A = 0},\; {B = 1},\; {C = 0}} & D_{{A = 0},\; {B = 0},\; {C = 1}} \\E_{{A = 1},\; {B = 0},\; {C = 0}} & E_{{A = 0},\; {B = 1},\; {C = 0}} & E_{{A = 0},\; {B = 0},\; {C = 1}} \\F_{{A = 1},\; {B = 0},\; {C = 0}} & F_{{A = 0},\; {B = 1},\; {C = 0}} & F_{{A = 0},\; {B = 0},\; {C = 1}}\end{bmatrix} \times \begin{bmatrix}A \\B \\C\end{bmatrix}}} & (6)\end{matrix}$

Where A, B, C are the color stimuli, e.g. CIE 1931 RGB color index; X,Y, and Z are the 3D locational address derived by Eq. (6); and theelements in the matrix of Eq. (6), i.e. D, E, and F, are convertingfactors. FIG. 5 schematically shows the generic structure of the presentnavigational device (502). When a properly constructed reference plane(510) is illuminated by light sources (503, 505, and 507) in differentcolors (e.g. R, G, B), the reference plane literally is “tinted” by therespective light sources. (Note that the light sources 503, 505, and 507can be placed anywhere in the navigation system; whenever the method oflight source placement is changed, the respective results shall changeas the natural consequence incurred by the relative position betweensaid respective light sources and reference surface 510). If the outputpower of the respective light sources are fixed, the tinting conditionof the reference plane will be denoted by a specific relative ratio ofthe intensity of light in different colors impinging on theoptoelectronic sensor (509) such as a CMOS sensor, i.e. the ratio R:G:B.We now use a color gamut to explain this in further detail. Referringnow to FIG. 15A, the locus that passes through points Q, R and S denotesthe full color gamut (e.g. at D65 lighting condition). As FIG. 5 shows,light sources (503, 505, and 507) are shining light at the planarreference surface (510) in different (e.g. red, blue, and green) colorsand, in general, from different angles (e.g. 0°, 120°, and 240°, but notshown here); the area on the reference plane (511) of which pictures aretaken by the sensor (509) within the disclosed navigator (502) can berepresented by a rectangular block (i.e. the range of the optoelectronicsensor), which is denoted as (1516) in FIG. 15A. Of course, when thegeometrical format of optoelectronic sensor (509) is not in rectangularshape, the shape of item (1516) will be changed accordingly. Thisrectangular block (1516) is then redrawn in FIG. 15B as plane (1507), onwhich every geographic point can be denoted by a unique color index data(e.g. CIE 1931 RGB). In FIG. 15B, we have picked out a point P′ todenote the origin of a 3D coordinate system having its three axespassing through point Q′, R′, and S′. Likewise, we have designated theaddress of the other three points on the reference plane to be Q″, R″,and S″. Since Point P′, Q′, R′, and S′ have different apparent colors,using Eq. (6), one is able to convert the respective color index datainto 3D positional data. Namely, we have picked out point (0, 0, 0),(a₁, 0, 0), (0, b₁, 0), and (0, 0, c₁) from a 3D orthogonal coordinatesystem. FIG. 6C shows that the mathematical plane on which cursormaneuvering occurs intersects with the axes of the coordinate system on(a₁, 0, 0), (0, b₁, 0), and (0, 0, c₁). Hence, the equation definingthis plane can be denoted in Eq. (7) as

$\begin{matrix}{{\frac{x}{a_{1}} + \frac{y}{b_{1}} + \frac{z}{c_{1}}} = 1} & (7)\end{matrix}$

Thus, when the presently disclosed navigator moves over a referenceplane tinted by light sources 503, 505, and 507, the address of allgeographical points lying thereon will satisfy Eq. (7). As FIG. 15Dshows, when a targeted object (1518A) is moved in a pixel frame (1507D),the pattern (e.g. square shape, with square block, with the twodiagonals designated by dark and bright pixels) of said targeted object(e.g. 1518E) will not be substantially changed by this motion. However,the relative intensity of light flows onto the respective pixels will bechanged (e.g. the signal level of the brightest pixel 1518F vs. that ofthe darkest pixel 1518E) in accordance with the relative distancebetween pixels and light sources. This phenomenon poses seriouschallenges for the prior art which uses block matching algorithms (BMA)to trace motions in that the BMA result will be different for 1518A, B,C, D. In the present invention, the illuminating sources (i.e. 503, 505,and 507) are mounted in the cavity on predefined positions (e.g. 0°,120°, and 240° around the targeted object). Hence, object 1518A, 1518B,1518C, and 1618D (actually they are the same object being pictured fromdifferent locations) will have different apparent colors in accord withtheir relative distances to the light sources of the navigator. Such adifference can be very delicate, but it does exist. Using a highly colorsensitive feature (e.g. high gain differential amplifiers), the presentnavigational device is able to determine the location of a targetableobject. When the navigator takes successive images of a targeted objectthat lies on the “tinted” reference plane, the trajectory of thetargeted object can be measured, and the hue value of said targetedobject can be measured as well. Hence, by analyzing the hue data of thetargeted object, the presently disclosed invention is able to determinethe location as well as motion (linear and non-linear) of the targetedobject with high precision.

When the tinting condition of the reference plane is changed, therespective point of intersection on the coordinate system will bechanged to, for example, (a₁, 0, 0), (0, b₁″, 0), and (0, 0, c₁). Again,as is denoted by FIG. 8C and Eq. (8), a new plane describes thiscircumstance (i.e., the plane representing the reference surface beingtinted by the above new tinting condition is denoted by a₁b₁″c₁).

$\begin{matrix}{{\frac{x}{a_{1}} + \frac{y}{b_{1}^{''}} + \frac{z}{c}} = 1} & (8)\end{matrix}$

As FIG. 15B shows, when the presently disclosed navigator designates aspecific 3D reference plane by a predefined tinting condition, it willbe able to move the cursor/object from point A′ (x₁, y₁, z₁) to B′ (x₂,y₂, z₂) on that plane, whose points of intersection with the three axesof the coordinate system are Q′, R′, and S′ respectively. In the priorart, the cursor maneuvering reference plane is considered only as a flatplane; there is no depth for the prior art's reference plane. As aresult of the surface tinting capability, when the presently disclosednavigator moves from point A to B, as a result of the variation oftinting condition, the z values change from z₁ to z₂; this denotes asituation where the depth value of the cursor/object has been changed.So, in the computer screen (1507D′) of FIG. 15C, the cursor appears asbeing moved closer to or away from the operator, corresponding to thenavigating act of the navigator (1502) on the reference plane (1513),which is in fact a 2D plane. In commonly assigned Ser. No. 14/056,140,Filing Date Oct. 17, 2013, a method is given by which rotationaldisplacement data ω can be measured from the mouse movement on a 2Dreference surface. Thus, the present system is able to use the teachingsof Ser. No. 14/056,140, Filing Date Oct. 17, 2013 to adjust thedirection of the cursor (604) (e.g. the one having an arrow shape inFIG. 6A); note that cursor (604) is always moving along the 2D plane(603A) in the 3D graphic rendering device (608A), unless the operatorwiggles his/her finger on the plane tilting angle adjust device (607) inFIG. 6B or unless the computer program itself adjusts the planeorientation of plane (603A) automatically.

When we want to tilt the cursor moving plane (603A), according to Eq.(3) the presently disclosed device has to provide a new set of unitvectors ({circumflex over (n)}=(n₁,n₂,n₃)) in order to derive therotational matrix R for Eq. (3). As FIG. 6C shows, in default condition,the unit vector {circumflex over (n)} is normal to the original cursormotion plane a₁b₁c₁. In practice, the normal vector {circumflex over(n)} can be acquired from Eqs. (7) or (8), or any 3D plane function(e.g. some data stored in the computer). Hence, while moving thenavigator body (607) on reference plane (606), the unit vector can beadjusted by the operator's finger wiggling action on the plane tiltingdevice (608) in FIG. 6B; the wiggling action of the finger allows theunit vector normal to the new cursor motion plane to not be normal tothe original cursor maneuvering motion plane (603A). This denotes asituation where the cursor/object is pointing at some direction out ofthe original cursor motion plane (1507). Under this circumstance, in the3D graphic rendering device (this can be illustrated schematically inFIGS. 8A and 9A), the operator will see the object/cursor (e.g. (804)and (904)) as being tilted by an angle in the space. From a mathematicalpoint of view, one is able to deduce that the original normal vector ofthe plane expressed by Eq. (7) is (a₁, b₁, c₁). When the plane istilted, the corresponding unit vector will be changed to (a₁, b₁″, c₁)per the case designated by Eq. (8). On the color gamut (1502), weunderstand that terminal points Q, R and S denote the three positionshaving the highest intensity of one color but the lowest intensity ofthe remaining two colors. In FIG. 15B, each of the three points Q′, R′,and S′ in the reference plane correspond to the three points that havethe highest coordinate values on each axis, (i.e. (S′, 0, 0), (0, Q′,0), and (0, 0, R′)). Thus, we will understand that point R′ is therepresentative of a color index data (R, 0, 0), where R in theparenthesis denotes the red color parameter (e.g. CIE 1931 RGB).Likewise, point Q′ is the representative of the color index data (0, G,0), whereas G in the parenthesis denotes the green color parameter. Bythe same token, point S′ is the representative of a color index data (S,0, 0). FIG. 15E shows that different 3D planes (e.g., (1519) and (1520))can be constructed by using different tinting conditions and they willbe characterized by different unit normal vectors (i.e., {circumflexover (n)}_(i), {circumflex over (n)}_(ii) where {circumflex over(n)}_(i) is the unit vector of plane a₁b₁c₁, and n, is the unit vectorof plane a₂b₂c₂). Once the above situation is established, the followingformulas (i.e. Eqs. (6A), (6B), and (6C)) can be expressed.

$\begin{matrix}{\frac{R}{R + G + B} = \frac{n_{1}}{n_{1} + n_{2} + n_{3}}} & \left( {6A} \right) \\{\frac{G}{R + G + B} = \frac{n_{2}}{n_{1} + n_{2} + n_{3}}} & \left( {6B} \right) \\{\frac{B}{R + G + B} = \frac{n_{3}}{n_{1} + n_{2} + n_{3}}} & \left( {6C} \right)\end{matrix}$

Now we refer back to Eq. (3). When the values of rotational angle ω, n₁,n₂, and n₃ are derived, the rotation matrix R can be determined.Furthermore, when the rotation matrix R is determined, as Eq. (2) shows,the rotational angles around x-axis, y-axis, and z-axis (i.e. ϕ, φ, θ)can be determined.

To an image sensor, hue value denotes photon flux in a specificwavelength (or a specific range of wavelengths) flowing onto the imagesensor. A conventional cursor maneuvering device does not have thecapability of using optical flow or hue value to measure motion vectors,or even the locations of objects. Thus, the present device exceeds theconventional art by its application of fundamental physics, i.e., ofoptical flows in different colors. In prior art (i.e. commonly assignedSer. No. 14/056,140, Filing Date Oct. 17, 2013), the navigator disclosedtherein is able to extract the rotational motion displacement from alumped 2D translational motion vector. We apply this technique to FIG.7B, and determine that the translational motion vector is (ΔX*, ΔY*) andthe rotational displacement as was disclosed by Ser. No. 14/056,140,Filing Date Oct. 17, 2013 is the label ω* in FIG. 7B. Hence, the presentinvention is able to maneuver an object from one point to another in a3D coordinate system by two different kinds of motions appliedconcurrently: translational and rotational motions; this has not beendone in the prior art (i.e. conventional optical mouse). The presentdevice is, therefore, a further significant advance of what is taught inSer. No. 14/056,140, Filing Date Oct. 17, 2013. By providing the abovestated unit axes (i.e. n₁, n₂, n₃), the presently disclosed navigator isable to move the cursor/object out of the original cursor motion plane(e.g. 1507 of FIG. 15B). In practice, while moving a cursor in a 3Dcoordinate system, the presently disclosed navigator crawls along agiven 2D reference surface, provided the unit vector {circumflex over(n)} is not changed. The value of n₁, n₂, and n₃ of Eq. (3) can beadjusted in more than one way (e.g. adjusted by the mouse buttons, givenby the computer program, etc.). It is also to be noted that both the 3Dtranslational motion vector (T_(x), T_(y), and T_(z)) and the rotationalmotion vector R in Eq. (1) are generated by one gestural movement of theoperator's hand (i.e. there is no need to decompose a motion intoseveral intermediate ones, like some of the prior arts require), thisunique feature matches the natural behavior of the human skeletalsystem, where the translational and rotational movements often takeplace simultaneously, by one gestural instruction given by the brain,with no motion decomposition. Thus, an important benefit offered by thepresent system is that the operator does not have to dissect a motion ofthe cursor by moving the navigator through several intermediate steps;this greatly reduces a barrier for new adopters in learning how to usethe navigation device and system.

It is to be noted that there is more than one way to determine the valueof ω of Eq. (3), the 3D rotational displacement data. In the firstmethod, one may use the method taught by commonly assigned Ser. No.14/056,140, Filing Date Oct. 17, 2013, use a block matching algorithm(BMA) to determine lumped translational motion vector(s), then decomposethe lumped translation motion vector(s) into a pure translational motionvector and a rotational motion vector. In a second way, the rotationalangle ω is determined by the second embodiment of the present system(see below). In either way, the rotational motion vector ω can be usedin combination with the unit vector (n₁, n₂, n₃) to derive therotational matrix R of Eq. (3). This matrix R offers great advantages tomodern 3D graphical rendering devices in that much of the calculatingprocesses on the computer, e.g. for deriving (n₁, n₂, n₃), or ϕ, φ, θ,can be eliminated. A conventional cursor maneuvering device cannot dothis easily.

From the above illustrations, it can be seen that the presentlydisclosed 3D navigation device is a high quality, even revolutionary, 3Doptical mouse having the ability to:

(1) Provide a 3D locational address in a format like (X, Y, Z).(2) Provide 3D rotational displacement data, like

$\begin{pmatrix}R_{11} & R_{12} & R_{13} \\R_{21} & R_{22} & R_{23} \\R_{31} & R_{32} & R_{33}\end{pmatrix}.$

(3) Fetch 3D data from a 2D reference plane.(4) Adjust the depth value of an object instantly, in a robust manner.

The modern optoelectronic image sensor is a 2D device that comprises aplurality of pixels within an image frame. Thus, the measurement ofmotion of an object on a reference surface is the result of measurementof light intensities in a pixel representation, and that measurementtakes two optical effects into account simultaneously: 1. the actualphysical motion of the object and; 2. optical artifacts that result fromthe variations of light flow (e.g. hue variations occurring duringmotion). The following is a brief presentation to indicate the roles ofthese physical and optical effects. We will refer to the navigatingdevice as an optical mouse, without loss of generality. We also recallthat the motion of an object being referred to is motion of that objectrelative to the moving mouse, i.e. the object may be some fixedparticles or other topological features on the surface of the mouse padover which the mouse is moving.

We assume that an object is moving in (or relative to) some space, e.g.,on a mouse pad, which is a 2D space, or in some surrounding ambient,which is a 3D space. Either will suffice for the following illustration.The image sensor is a 2D optoelectronic device comprising a plurality ofpixels. Hence, after a complete image capturing process (e.g. a shutteropening and closing), we can denote the respective pixel data by atime-dependent light intensity function I (x, y, t). When the objectmoves, the image sensor senses a corresponding motion vector (velocity)m in the pixel plane, which is denoted as

{dot over (m)}=[x,y]^(T)  (7)

Thus, we have

$\begin{matrix}{v_{m} = {\overset{.}{m} = {\left\lbrack {v_{x},v_{y}} \right\rbrack = \begin{bmatrix}{{dx}/{dt}} \\{{dy}/{dt}}\end{bmatrix}}}} & (8)\end{matrix}$

Assume m is unchanged in the time interval dt, that is,

I((x+v _(x) dt),(y+v _(y) dt),(t+dt))=1(x,y,t)  (9)

Again, I(x, y, t) denotes the intensity of light as is detected by aphoto-sensor lying at the position (x, y) in a 2D image frame at time t.

If the illuminating source of an optical mouse (e.g. an LED cluster)projects three light beams, each in a color different than those of theothers (e.g. R, G, B), and the image sensor embedded in that sameoptical mouse is color sensitive to each of the colored light beams(i.e. it can tell the differences among the light beams in differentcolors), then Eq. (9) can be re-written as Eqs. (10A), (10B), and (10C).

I _(RED)((x+v _(x) dt),(y+v _(y) dt),(t+dt))=I _(RED)(x,y,t)  (10A)

I _(BLUE)((x+v _(x) dt),(y+v _(y) dt),(t+dt))=I _(BLUE)(x,y,t)  (10B)

I _(GREEN)((x+v _(x) dt),(y+v _(y) dt),(t+dt))=I _(GREEN)(x,y,t)  (10C)

Where I_(RED)(x, y, t) denotes the intensity of red light as detected bya photo-sensor lying at the position (x, y) in a 2D image frame at timet, I_(BLUE)(x, y, t) denotes the intensity of blue light as detected bya photo-sensor lying at the position (x, y) in a 2D image frame at timet, and I_(GREEN)(X, y, t) denotes the intensity of green light asdetected by a photo-sensor lying at the position (x, y) in a 2D imageframe at time t, respectively.

Note that the above three optical flow equations (i.e. 10A-10C) areindependent from one another (i.e. the signals are non-interfering).When the intensity of the red light beam changes, it will not affect thereadout data of the blue and green optoelectronic sensor, and viceversa.

When the intensities of the respective light beams vary with x, y, andtin a very smooth way, Eq. (9A-9C) can be expanded in a Taylor series asEqs. (11A-11C) below.

$\begin{matrix}{{{I_{RED}\left( {x,y,t} \right)} + {\frac{\partial I_{RED}}{\partial x}v_{x}{dt}} + {\frac{\partial I_{RED}}{\partial y}v_{y}{dt}} + {\frac{\partial I_{RED}}{\partial t}{dt}} + {H.O.T._{RED}}} = {I_{RED}\left( {x,y,t} \right)}} & \left( {11A} \right) \\{{{I_{BLUE}\left( {x,y,t} \right)} + {\frac{\partial I_{BLUE}}{\partial x}v_{x}{dt}} + {\frac{\partial I_{BLUE}}{\partial y}v_{y}{dt}} + {\frac{\partial I_{BLUE}}{\partial t}{dt}} + {H.O.T._{BLUE}}} = {I_{BLUE}\left( {x,y,t} \right)}} & \left( {11B} \right) \\{{{I_{GREEN}\left( {x,y,t} \right)} + {\frac{\partial I_{GREEN}}{\partial x}v_{x}{dt}} + {\frac{\partial I_{GREEN}}{\partial y}v_{y}{dt}} + {\frac{\partial I_{GREEN}}{\partial t}{dt}} + {H.O.T._{Green}}} = {I_{GREEN}\left( {x,y,t} \right)}} & \left( {11C} \right)\end{matrix}$

Where the H.O.T. stands for the “higher order term(s)”, e.g. a

$\frac{d^{2}}{{dt}^{2}}$

term. For example, when there is a rotational movement of the mousebody, the

$\frac{d^{2}}{{dt}^{2}}$

term is no longer a term that is negligible. We begin with the situationwhere the non-linear term is negligible. Assuming that the brightness(i.e. reflectivity) of the reference surface does not change abruptly,Eqs. (12A-12C) will describe the behavior of each kind of pixel (e.g.red, blue, and green) that is receiving its radiation.

$\begin{matrix}{{{\frac{\partial I_{RED}}{\partial x}v_{x,{RED\_ PIXEL}}} + {\frac{\partial I_{RED}}{\partial y}v_{y,{RED\_ PIXEL}}} + \frac{\partial I_{RED}}{\partial t}} = 0} & \left( {12A} \right) \\{{{\frac{\partial I_{BLUE}}{\partial x}v_{x,{RED\_ PIXEL}}} + {\frac{\partial I_{BLUE}}{\partial y}v_{y,{BLUE\_ PIXEL}}} + \frac{\partial I_{BLUE}}{\partial t}} = 0} & {\left( {12B} \right)\;} \\{{{\frac{\partial I_{GREEN}}{\partial x}v_{x,{RED\_ PIXEL}}} + {\frac{\partial I_{GREEN}}{\partial y}v_{y,{GREEN\_ PIXEL}}} + \frac{\partial I_{GREEN}}{\partial t}} = 0} & \left( {12C} \right)\end{matrix}$

In differential geometry formulation, equations (12A˜12C) can be writtenas:

$\begin{matrix}{{{{\nabla I_{RED}} \cdot v_{mRED}} + \frac{\partial I_{RED}}{\partial t}} = 0} & \left( {13A} \right) \\{{{{\nabla I_{BLUE}} \cdot v_{m,{BLUE}}} + \frac{\partial I_{BLUE}}{\partial t}} = 0} & \left( {13B} \right) \\{{{{\nabla I_{GREEN}} \cdot v_{m,{GREEN}}} + \frac{\partial I_{GREEN}}{\partial t}} = 0} & \left( {13C} \right)\end{matrix}$

Where

${\nabla I} = \left\lbrack {\frac{\partial I}{\partial x},\frac{\partial I}{\partial y}} \right\rbrack^{T}$

is the image gradient at pixel [x, y].

Based on the principles of optical flow physics, when multiple lightbeams impinge on an image sensor (e.g. I_(RED), I_(BLUE), and I_(GREEN)in Eqs. (13A˜13C)), a plurality of optical flows are produced. Note thatan optical flow can be generated by either physical objects or opticalartifacts (e.g. light beams in different colors, shadows, etc.). Thus,the image sensor is able to take pictures of physical objects (e.g.particles on the reference surface) as well as the hue variations of thetinted reference surface. In the situation created by the presentdevice, three kinds of colored light rays (i.e. light of differentwavelengths) impinge on the reference plane. More than three kinds canbe used, but are generally not necessary. After reflection, the colorimage sensor embedded in the cavity of the navigational device willdetect three “hue flows” on its pixel plane (where “hue flow” designatesan optical flow per a specific wavelength of light). Of course, thedevice can use alternative methods, such as light beams in threedifferent polarities (polarizations), UV or IR light, lasers, differentphases, etc. to obtain similar results. Methods as these are essentiallyusing the same spirit taught by the present invention.

If we investigate these optical flows further, we will discover thateach of them can produce an image gradient in accordance with a hueprofile (hue variations along the surface) that is largely determined bythe tinting condition of the reference surface (or whatever surfacetreatment is used to affect polarizations, etc.) as well as the spectraldistribution of the respective light sources. Thus, using Eqs.(13A-13C), one can derive three sets of motion vector data from thenavigator.

As was stated above, the planar reference surface used by the navigatorhas been “tinted” with three colors (so, we may call it a colored mousepad), which can be done by physical means, chemical means, or opticalmeans. As a result of the light reflectance property of the tintedreference plane, the reflected light beams from the reference planeproject three independent optical flows onto the image sensor. Sinceeach light flow represents a predefined image gradient in a specificcolor (or, using alternative methods, polarization, phase, etc.), thethree color index data (e.g. CIE 1931 RGB) of the colored referenceplane (i.e. the mouse pad) will vary from place to place independently.As a result, the color-sensitive image sensor in the navigator is ableto detect the variations of color index in an effectively “orthogonal”(i.e., independent) manner, so that the orthogonal color index data areall mapped to the geographical position of the reference plane. As aresult of this unique capability, the present system is able toestablish a “geographical map” based on the color index data acquiredfrom the “tinted” reference plane. The color index (e.g. CIE 1931 RGB,etc.) serves as an ideal data source by which the navigator canascertain its absolute whereabouts on the “tinted” reference plane.Thus, the positional data [X, Y, Z] of Eq. (1) can be determined. Thenavigational device may still use metrical methods (e.g. BMA) to traceobjects on the reference plane (e.g. particles, surface morphology orother topological features); these microscopic objects are so small thatthey will not affect the apparent color of the reference plane and theycan be used independently and simultaneously to obtain relative motionvectors. As NU11-007 teaches, the navigator is able to measure thetranslational and rotational motion vectors by the relative movementbetween the navigator and the targeted objects lying on the referenceplane. In addition to the method used to convert the hue data to the 3Dpositional data, the present navigator is able to determinetranslational motion vector [T_(x), T_(y), T_(z)] and rotational motionvector w concurrently through surface features.

We would now like to highlight the importance of positional accuracy toimplement a graphical design rule that is vastly different than that ofthe conventional art: the physical alignment of the navigator andgraphic rendering device. This is an issue that has been largely ignoredin the prior art. When the optical mouse was invented, there was no needfor the operator to align his/her hand movement to that of the cursor onthe screen. Today, as graphical rendering technology becomesincreasingly used in a wide variety of applications, a mouse can be usedby a graphic artist or a surgeon. People in these professions rely onfinesse to perform their jobs, so the mouse they need is one that alignstheir hand movement to that of the cursor on the screen. If one usesEqs. 11A through 11C to understand these new requirements, it impliesthe H.O.T of each equation can no longer be ignored. By extensivepractice, for example, an operator who is a cartoon artist can become sofamiliarized with the effects of his/her hand movement (comprisingmostly linear terms and H.O.T.) in a predefined coordinate system, thatthe “creature” he/she generates can be effectively “mapped” onto thegraphics system by the neural functions in the operator's brain.

In graphic work, typically the renderings from the same artist may havean identifiable “genre”, created by the delicate gestural movement (i.e.precisely the H.O.T. of Eqs. 11A-11C) of the artist. Today's motiontrackers (both 2D and 3D) have ignored the importance of these H.O.Ts increating such identifiable subtleties and nuances of an artisticrendering.

The present system not only performs metrical comparisons using BMA, butalso measures hue variation. In a targeted pixel block, the variation ofhue corresponds to the H.O.Ts of Eqs. 11A, B, and C. Thus, thenon-linear motion of the operator's hand can be used for delicate motioncontrol of the computer cursors. Because of the way the presentlydisclosed navigational system “tints” the reference surface, eachgeographical point is allowed to reflect light to produce a unique colorindex parameter; when taken together, all these color index datagenerate a 3D space. This is the first time in the industry that anavigator and its planar reference surface have a built-in coordinatesystem. In the past, the graphical object, (e.g. a cartoon creature) asseen in an animated art movie, is generated by mathematical programsinstead of an artist's hand. As a consequence, today's digital cartoonhas disengaged itself from the genuine creativity of the artist. As aresult of the upcoming revolution of 3D graphic art, an imminent demandwill emerge for the electronic industry to develop a new navigationdevice that “aligns” the data streams between two “central processingunits”, the computer's CPU, and the human brain of the artist/operator.In the process of delivering motion vector data in the computer, thepresently disclosed navigator provides electronic signals for thecomputer. Simultaneously, a similar process takes place in theoperator's brain; it is the nervous system of the operator thattransmits the positional data of the presently disclosed navigator backto the brain. The latter operation is not explicit to human eyes(especially the H.O.T., the non-linear motions). But indeed the H.O.T.has much to do with how the smoothness and agility of motions aretransmitted to the rendering device. When an artist uses the presentlydisclosed navigator, through familiarization by repetitive practice, theindividually recognizable quality of his/her work can be demonstratedand visualized through subtleties of the sketching, calligraphy,drawing, etc. In the past, it was exactly these artistic nuances thatwere responsible for bringing human civilization into modernity. In mostof the hand-held navigators we find in the mass market today, the abovedescribed product design concept is often confused with “ergonomics”.But that term refers only to ease of long term use and energy expendedby the user. It does not adequately describe the brain-to-handtransparency provided by the present device. Today a consumer can easilyfind some so-called 3D motion tracker in the electronic market. But thatproduct may not necessarily have the key feature of alignment as statedabove, which is required by the human brain for subtle, nuanced andsophisticated work (e.g. surgery, painting, etc.). There are some motiontrackers that use optical means, magnetic means, electro-mechanicalmeans, or electromagnetic (EM) means. These products still ignore theissues of alignment; almost all of them have a generic drawback, thenatural vibration caused by human hand/body has not been addressed, andthe non-linear portion of operator's hand cannot be measured reliably.FIG. 13A shows the vibration model of a human body being seated, withone hand gripping an object (e.g. a motion tracker). The entire skeletonof human body is like a mechanical frame bolted together by manysprings. FIG. 13B shows another vibration model for a human body beingin the standing position. The magnitude of the vibrations will changewhen the attitude and gestures of the human body changes. Industrialsurveys reveal that difficult gestures yield even larger positionalerrors than the easy ones. When an operator uses a 3D motion tracker asshown in FIG. 11 to maneuver an object in the displaying device thatrequires high positional accuracy (e.g. medical images, mechanicaldrawings, etc.), these 3D motion trackers may fail to meet the stringentrequirement of sophisticated applications. As of such, although thereare some 3D motion trackers being sold in the mass market (e.g. NINTENDOWii, Microsoft Kinect, etc.), their applications have been largelylimited to video games or other applications that do not require highpositional accuracy. The presently disclosed navigator is mechanicallysupported by a reference surface, so the influence of hand vibration onsaid navigator can be suppressed; this makes the present navigator oneof high sensitivity and high resolution.

We have cited three major advantages of the present system in the aboveparagraphs:

1. deriving 3D motion vector data from a 2D reference plane,

2. providing absolute addresses in high sensitivity and high resolutionmode for sophisticated 3D applications, and

3. the ability to detect non-linear motion reliably.

A fourth advantage of the present invention has to do with scaling. Ifan optical mouse has a scaling capability, it is equivalent to sayingthat the image captured by the optical mouse has a “depth of view” or“depth of field” on the reference plane. Expressed mathematically, forthe same amount of displacement on the planar reference surface,depending on the scaling factor, the positional readout of the presentlydisclosed navigator will be made to vary. In conventional art, as FIG. 1shows, the same geographical point on the reference surface (i.e. themouse pad) may only yield one 2D address (x, y). In the present system,because the 3D data can be controlled by the hue data; the same positioncan yield different 3D address. As FIGS. 10A and 10B show, the depthvalue of cursor (1004A) can be changed from literally zero to minusinfinity (−∞) when the cursor moves from the address of point (1004A) tothat of point (1001A). On the reference plane (1006), it denotes a path(not shown) that comes across a unique point, which corresponds to whatwe call the point of minus infinity or the vanishing point. At thispoint all objects will appear as a dot. Conventional cursor maneuveringdevices do not have this feature; they do not have a vanishing point(although the 2.5D or 3D graphic rendering devices do) within the scopeof device movement. Limited by such a shortcoming, the range of movementused by today's motion tracker on its reference surface is not alignedwith the corresponding movement of the cursor displayed in thedisplaying device. The present system uses three optical flows to denotethe absolute address on the reference plane. Hence, the present systemis able to designate a point on the reference plane as the vanishingpoint in the 3D graphic display. While the navigator is maneuvering onthe reference plane, the z value changes in accordance with variation oftinting condition of said reference plane at all times. Though handmaneuvering actions, the above feature allows the operator to gain afeeling that he/she is navigating an object in a 3D space that has sixdegrees of freedom: +X, −X, +Y, −Y, +Z, −Z.

The cursor maneuvering devices used by the consumer electronic industrytoday is well behind what the system invention offers. There are manyoccasions where even a simple movement as shown on a 2.5D or 3D devicerequires multiple movements of the cursor maneuvering device (e.g.zooming in and out). Once a simple movement of a displayed object has tobe dissected into multiple steps for a cursor maneuvering device, thehuman brain may easily lose valuable clues with which to analyze themotion he/she sees in the displaying device. As was previously stated,such an analytical capability of human brain is the origin ofcreativity, an essential value of modern human civilization.

“Vanishing point” is not just a phrase used by graphic artists; it hasto do with the professional work associated with the computer orelectronic system that has a graphic displaying device (e.g. 3Dgraphics, surgery robot, etc.). Perspective in a sketch adds realism tothe three-dimensional representation. The fundamental rule for aperspective sketch is that parallel edge lines converge to a singlepoint (the vanishing point) on the horizon. If there is just onevanishing point, then this sketch is called a one-point perspectivesketch. If two sets of parallel edge lines converge to their respectivevanishing points, then it is called a two-point perspective sketch.There is also case of three-point perspective, in which all verticallines converge to a point. The present system relies on three ways tomanipulate the location of a vanishing point, i.e. the tinting conditionof the reference plane, the illuminating condition of the light source,and the image capturing process scheme (e.g. shutter speed, duration,etc.). Conventional cursor maneuvering devices do not provide such avanishing point on their reference medium (e.g. mouse pad); theoperator's hand gesture cannot be directly aligned to the coordinatesystem displayed by the graphic rendering device that has vanishingpoint(s). An ideal 3D cursor maneuvering device provides an element of“finesse,” by allowing the operator to coordinate his hand and eyemovement by means of a coordinate system that “aligns” the scenery thathe sees on the display to what his hands are doing on the mouse pad. Theabove feature establishes the future fundamental difference between aprofessional navigator and a commoditized one.

Based on the above illustration, we understand that today's electronicindustry needs a navigation device that not only provides 3D orthogonalpositional data (i.e. in (x, y, z) format) with high positionalaccuracy, but also a device that is capable of designating at least onevanishing point on the reference plane/space. With all these featuresincluded in one embodiment of such a navigation device, there is oneadditional feature needed for high sensitivity and high resolutionpurposes: the navigational device must be immune from human bodyvibration.

The present navigation device has two working modes, 2D and 3D. The datagenerated in one mode can be correlated to the one generated by theother mode. When working as a 3D cursor maneuvering device, as schematicFIG. 15C shows, the color sensitive feature of the 3D optical mouse(1512) will be activated and a tinted mouse pad (1613) will be used inconjunction with the resulting color sensitive optical mouse (1612).During its operation in this mode, the 3D cursor maneuvering device(1512) shines multiple light beams of different wavelengths onto thetinted mouse pad (1513). These multiple beams may irradiate a targetedobject from different directions. A color image sensor (not shown inFIG. 15C) embedded in the optical mouse (1512) is sensitive to the lightbeams in their respective wavelengths, which are transmitted back to thesensor (e.g., by reflection). Hence, the tinted mouse pad (1513) and theoptical mouse (1512) work in collaboration as a 3D cursor maneuveringsystem.

When placed on an ordinary desktop surface instead of the tinted mousepad (1513), the same cursor maneuvering device (1512) works as a 2Doptical mouse. Selection of the 2D mode can be made manually orautomatically. It must be noted that regardless of whether its 3D or 2Dmode is being used, the cursor maneuvering device (1512) itself (not thecursor on the displaying device) only moves on a two dimensional plane(e.g. the desktop surface or tinted mouse pad). This unique capabilitygives an extraordinary advantage to its operators, who can easily usethe navigational device in a professional office environment, since theactual motion of the device remains the same in both modes. Theconventional 3D trackers (e.g. the stick (1105) shown in FIG. 11) arenot user-friendly within the office environment.

The color image sensor embedded in the 3D optical mouse (1512) alsooffers two other unprecedented advantages, high resolution and highsensitivity. A modern digital image sensor can easily capture thousandsof images in one second; the number of pixels in an image sensor caneasily be in the millions, although such an amount is not required bythe present system. The present navigational device provides theresolution and high sensitivity 3D cursor maneuvering to take advantageof these properties. There is yet one more benefit that accompaniesthese high resolution and high sensitivity features: a perspectivesketching capability. This seemingly easy but actually difficult methodof sketching is extremely important to next generation 3D graphic artrendering devices or 3D printer (some reasons have already been given,others will be explained in the latter sections); yet it has not beenaccomplished by the prior art.

As we have already noted above in our discussion of the vanishing point,perspective sketching is the foundation of modern graphical art. In aperspectively sketched drawing, objects located in remote space aredrawn smaller than the ones that are located closer to the viewer.Hence, referring to FIG. 6A, within the image frame (608A), object(602A) will appear larger than object (604A), despite the fact that(604A) and (602A) are actually two objects with the same physical sizeand shape. This rule of sketching can be applied to the whole picture.When an object is placed at infinity, its apparent contour will shrinkto a dot (e.g. point 601A). This point is the vanishing point that wasalready discussed; all objects in the image frame (608A) will now besketched in such a way that their horizontal contour lines (e.g. lines(605), (605′) and (605″)) merge at a vanishing point (601A). The abovestated technique, which defines perspective sketching, was developed inor about the time of the Italian Renaissance era. Through centuries ofsubsequent development, this technique has become a mature and commonmeans of sketching which is used by almost all digital graphicalrendering devices today (e.g. flat panel displays having 3D graphicrendering capability). On the other hand, over the years, cursormaneuvering devices have not developed as much; what most people use isstill a 2D optical mouse. The presently disclosed navigator devicecloses this gap.

To illustrate the merit of the presently disclosed navigator with regardto said perspective sketching capability, an exemplary scenario isdepicted in FIG. 16A, in which several 3D objects (i.e., object1, 2, and3) are placed in a world coordinate system W, which is further beingperspectively perceived by a camera coordinate system Q. Let {rightarrow over (Q)} be the vector denoting the position of a point Q withregard to said world coordinate system, and The a vector denoting theposition of a targeted object (say, object 1) with regard to the cameraQ. From basic vector algebra, one may find that the unit vector of saidcoordinate system of world, which can be denoted as {right arrow over(W)}, shall comply with Eq. (14).

$\begin{matrix}{\overset{\rightarrow}{W} = \frac{T - Q}{{T - Q}}} & (14)\end{matrix}$

If a device engineer has a mean to imprint the world coordinate system(denoted by its unit vector {right arrow over (W)})) onto his mouse pad(in other words, the coordinate system W′ is only meant for theperception of the presently disclosed cursor maneuvering device, or foroperator's hand to “feel”, but not for human eyes to perceive), then allthe objects (object 1, 2, and 3) can be literally transcribed from thememory of computer or electronic system to said mouse pad (e.g., 1601A).Thus, as FIG. 16A shows, it is interesting to notice that now both thecamera Q and mouse pad (1601A) can have their own 3D coordinate systems,on which there lies their own vanishing points (not shown in FIG. 16A,shown in FIG. 16B).s FIG. 16B shows an exemplary case that the entirescene of a graphic rendering device (not shown) has been “trans-printed”to a mouse pad (1608B). The readers must be specially advised that saidtrans-printed scene on said mouse pad (1608B) is an imaginary one—it maynot have the color or contour as our eyes normally perceive. All thepresent invention means here is that each of the geographic pointsthereon can find a corresponding one on the graphic displaying device,and vice versa. Hence, point (1601B) shall denote the vanishing point ofthe reference plane (1608B). One may notice that some contour lines ofthe objects also get their corresponding sketching lines trans-printedon the mouse pad (1608B), i.e., (1602B) and (1603B). These “sketchlines” may be visible or invisible to human eyes. What these lines tellus is that the reference plane (i.e., mouse pad 1608B) can provide 3Ddata in accordance with a pre-defined perspective angle. In certainsituations, however, the perspective angle must be adjusted in a timelymanner. As an exemplary case of using the presently disclosed cursormaneuvering device to adjust the perspective angle, FIG. 16C shows thatthe operator maneuvers his/her finger (1604C) on the touch sensingfeature (1605C). When the finger (1604C) is moved to the left of thecentral line (1606C), said touch sensing feature (1605C) willacknowledge the computer or electronic system to adjust the perspectiveangle accordingly. Almost simultaneously, as FIG. 16B shows, the“imaginary” vanishing point (1601B) of the mouse pad (1608B) has beenmoved leftward as compared to that of FIG. 16D. By the same token, asFIG. 16E shows, when the operator's finger (1604E) moves to the right ofthe central line (1606E), the position of the vanishing point (1601D) onthe mouse pad (1608D) changes accordingly. What the presently disclosedcursor maneuvering device really do is that, when the computer orelectronic system desires to change a perspective angle, it acknowledgesthe presently disclosed cursor maneuvering device. By providing thedisplacement of operators' finger on the touch sensing device (1605C orE), a perspective angle moving instruction plus the corresponding datais sent to the computer electronic system for the above stated purpose.In the meantime, the presently disclosed cursor maneuver device cancalculate the new address for said vanishing point (i.e., 1601B or D) onsaid tinted mouse pad (1608B or D). Accordingly, a new set of hue datacan be determined for said process. It is now clear that a “tinted”mouse pad as is disclosed by the present invention plays the vital roleto the next generation 3D perspective computer graphics. Note that theabove processes can all be done by the wiggling act of the operator'sfinger instead of moving the body of the presently disclosed cursormaneuvering device (e.g., 1607C, 1607E). When an operator moves thepresently disclosed cursor maneuvering device (e.g., 1607C, 1607E) onthe tinted mouse pad (e.g., 1601A), the situation can be verydynamic—there are 3D motions for the objected selected as well asperspective angles adjustment, these incidents may take place by onegestural movement of operator's hand/finger (although the internal clockof the presently disclosed cursor maneuvering device may separate thetasks into different working durations). In one possible scenario, boththe object (e.g., object 1, object 2, and object 3 in FIG. 16A) and theworld coordinate system W shall move. In another, the object is movedfirst, and a world coordinate system change takes place shortlyafterwards.

Professional artists are trained to draw objects on the sketch paper inaccordance with the rules of perspective sketching. Since the sketchingpen itself does not have a coordinate system, the artists have to relyon a lot of practices on the sketching paper to familiarize his handmovement with the 3D perception that he desires to put on paper, just asFIG. 17 shows. When an artist moves his/her work place to the computer,his/her brain controls the neurons that links to eyes and hands. Thebrain can construe neuron signals from the eyes and hand (fingers,wrist), and the come out with some curves, figures, patterns, etc. Theabove stated cartoon character Ariel is an example. In practical world,there is no such a mermaid at all. So, there comes a demand from thegraphic artist for a 3D cursor maneuvering device that combines theartist's will and hand gesture movement to render a beautiful cartoon.As was stated earlier, this kind of demand did not exist at the time theoptical mouse was just invented, it only become an imminent demand afterthe graphic rendering technologies had reached a certain level ofmaturity in about the last decade.

Because the presently disclosed system effectively carries a 3Dcoordinate system on the tinted mouse pad, it offers a genuine ergonomicadvantage that prior art does not have, particularly from the point ofview of an artist. FIG. 17 is the sketch of a cartoon girl. The viewermay quickly get a perception that this girl is beautiful, having a goodfacial structure despite some of her details are still missing (e.g. eyeballs, lips, etc.). To infuse an aesthetic feeling to a graphical objectas such, the strokes of the artist (note they are in 2.5D coordinatesystem) play the vital roles. Hence, it is the alignment process betweenthe mouse operator's eyes and hand that matters. This is contrary to theconventional optical mouse whose design is focused on aligning itspositional accuracy with the graphic displaying device. As has beenstated above, generating a 2.5D stroke on a graphic displaying device isnot difficult, generating a 2.5D stroke by a navigator that allows theoperator to analyze the result in the displaying device and adjusthis/her hand to generate a stroke that represents his feeling better inthe next trial is difficult. To sketch just one stroke beautifully, anartist has to practice so that his/her hand movement gradually respondsto the aesthetic feeling in his/her brain. As the rough sketch in FIG.17 shows, the girl's hair has been drawn as several separate lines(1701) and (1702), each sketch line represents a trial process of theartist to produce the final acceptable affect. During the course of haircurve sketching, the artist gradually familiarizes/aligns his/her handmovement to what his brain wishes to convey. To support an artist who isattempting to do the same on a computer, the navigational device musthave an “implicit” coordinate system, one that is not actually visibleto the eye, that the artist can “memorize”. In most cases, the computerdoes not know the existence of such a coordinate system. But when theoperator moves the navigator on its reference plane, its existence willbe clearly felt in terms of the result. Thus, throughout the entiremaneuvering process of the presently disclosed navigator, a 3Dcoordinate system is always projected onto the “envisioned” sketch paper(i.e. FIG. 17), with or without the artist's acknowledgement byeyesight.

As has been stated above, the computer industry has entered an era inwhich the cursor maneuvering device must be capable of perspectivesketching. The capability of taking data from a glossy surface may nolonger be the key performance metric. In modern digital graphic artutilities, the “place” where the artist's sketch pen swiftly fliesacross the paper corresponds to the reference plane of the cursormaneuvering device, i.e. the mouse pad. Conventional cursor maneuveringdevices (e.g. the roller ball type, turning knob type, or optical type,etc.) do not have such a 3D reference plane; there is no direct means tomap the respective points in the reference medium to the pixels in the3D rendering devices, and, thence through the artist's neurons, back tohis/her human brain.

We learned from the history of art in human civilization that an ideal3D cursor maneuvering device should allow the operator to control hishigher order hand movements in a 3D coordinate system that lies on a 2Dplane, so that hand vibration, jittering, etc. can be suppressed. Such a3D coordinate system can be mapped by its user to (or between) two imageframes, the image frame associated with the computer, and the imageframe that is created by a plurality of “neuron signals” that are in theuser's brain. These are fundamental characteristics that differentiatethe present system from the prior art. To reiterate, what we refer to asa reference “plane” need not necessarily be a 2D space. As professionalartwork since renaissance times has demonstrated (FIG. 17), a 2Dreference plane (i.e. the canvas or sketching board) can provide arealistic 3D perspective rendering effect to human eyes, so long asthere is a stable vanishing point on it (much like point (1601B) is tothe mouse pad (1608B)). Thus, we come to the realization that areference plane that carries the “absolute” positional address of therespective geographic points thereon, or any means/device withequivalent effect, is a key feature of the next generation 3D navigationdevice. And that is precisely a feature of the present navigationaldevice.

The merits of using a 2D reference plane with a vanishing point on itcan be demonstrated by FIG. 18 and FIG. 19. FIG. 18 is a cartoon made byhand sketching. FIG. 19 is a cartoon created by a digital graphic artrendering devices (e.g. graphic processor, such as NVIDIA®'s graphicprocessor). As FIG. 18 shows, the sketcher used different curves topresent the 3D gesture of the dancers. FIG. 19 does not have thischaracteristic. Although the general appearance of the cartoon in FIG.19 is sleek, modifying this cartoon object (FIG. 19) in accordance withthe artist's desires is not easily done as there is no coordinate systemfor the mouse used. If a graphic artist desires to change the gesture ofthe rabbit in FIG. 19, then the entire graphic art rendering system mustgo through a complicated process comprising object database management,vertex mapping, texture mapping, polygonal mapping, and shadowgeneration, etc. For a graphic artist, creativity is often developed bythe alignment process between eyes and hands rather than by thegeneration of mathematical curves (most of the curves in FIG. 19 aregenerated by mathematical means). If a cursor maneuvering device doesnot support the artist in his task of directly relating his/her handmovement to the results produced, then the quality of the work will belimited by a lack of spontaneity. In other words, the artisticcreativity, the most valuable and human part of a work of art, has beensacrificed by the shortcomings of the conventional cursor maneuveringdevice (both 2D and 3D types). Thus, there is a need for the electronicindustry to develop a 3D cursor maneuvering device that facilitatesperspective sketching.

Of course, there are still many operations in the real world thatrequire not the operator's creativity but the device's resolution. Forwork that requires high resolution (e.g. AUTOCAD® drawing, medicalimages), the presently disclosed navigator also has the requiredcapabilities. These are attributed to the hue images of the targetedmicroscopic objects on the desktop surface. The apparent color of anobject (e.g. a dust particle) can be changed by using different incidentangles of the variously colored light rays (note that when a mousemoves, the relative position of the targeted object and the image sensorembedded in the mouse changes, consequently the incident angle of therespective light rays of different colors changes). Calculating thetranslational motion vector as well as the rotational one by the methodtaught herein is far easier, from an algorithmic point of view, and moreprecise, as compared to that of prior art.

BRIEF DESCRIPTION OF THE DRAWINGS

This disclosure will be described with reference to the accompanyingdrawings, wherein:

FIG. 1 schematically depicts the structure of a conventional opticalmouse and an associated computer; the mouse only provides twodimensional relative motion vectors (i.e. no absolute addresses) for thecomputer to use.

FIG. 2 shows, schematically, an artist's rendering of an airplane modelin a “2.5D” coordinate system, i.e., each of the respective points onthe airplane fuselage has a depth value and there is a separaterotational axis.

FIGS. 3A and 3B show, schematically, how a 2.5D coordinate systemdepicts a 3D object on a planar displaying device.

FIG. 4 schematically shows the present device being manipulated to makeuse of the plane-tilting function.

FIG. 5 schematically shows a cross-sectional view of the present device,such as the device being manipulated in FIG. 4.

FIG. 6A-6C shows, schematically, how a cursor can be made to move alonga plane of motion in a 3D coordinate system by the present navigationalsystem.

FIG. 7A-7C schematically shows how a cursor can be made to rotate aplane of motion that lies in a 3D coordinate system.

FIG. 8A-8C schematically shows how cursor motion can take into account achange in vanishing point.

FIG. 9A-9C shows schematically additional motions of the cursor of FIGS.8A-8C.

FIGS. 10A and 10B schematically shows how the present device can fostercollaborative actions between an operator's fingers, wrist and handmovements to manipulate a moving cursor.

FIG. 11 schematically shows an EM tracker used in the prior art to trackan operator's hand motions.

FIG. 12 schematically shows how a camera is used to track an operator'shand motions.

FIG. 13A-13B schematically depicts a mechanical model of vibration of aseated human (A) and a standing human (B).

FIG. 14A schematically depicts a “playroom” of the future in whichdifferent surfaces will be used to guide the motions of a navigationaldevice.

FIG. 14B schematically shows three designs of the present navigationaldevice that include a zoom in/out feature.

FIG. 14C-14D schematically show the effects of a cursor that can bothmove 3-dimensionally and still exhibit the features of a 2D device.

FIG. 15A-15F schematically depicts an exemplary color gamut on a tintedplanar reference surface and the formation of a corresponding 3Dcoordinate system that can be represented (i.e., mapped into) by thisgamut using the present system.

FIG. 16A schematically depicts objects in a scene drawn using theperspective feature of the present system and showing lines that areextrapolated to a single vanishing point.

FIG. 16B-16E schematically depicts further applications of perspectivedrawing using a vanishing point.

FIG. 17 schematically shows an artist's hand sketch of a model (a 3Dobject) in which a 2.5D coordinate system must be inferred by theartist.

FIG. 18 schematically shows a hand sketch of a model in motion, showinghow the artist's use of subtle nuances gives the impression of motion.

FIG. 19 schematically shows a 3D model whose shape is provided bymathematically calculated functions rather than an artist's handgestures.

FIG. 20A-20F schematically show a tinted planar reference surface whichis able to provide three non-interfering profiles that will produceabsolute 3D positional data for the computer to use.

FIGS. 21A and 21B schematically illustrates how a 2.5D graphicalrendering device uses a 2.5D reference plane when using the presentdevice.

FIG. 22 schematically shows a 3D touch pad using multiple light sourcesas an alternative implementation of the present device.

DETAILED DESCRIPTION

Referring to schematic FIG. 15C, the presently disclosed 3D navigatingsystem is shown as consisting of a movable device (1512) (alternativelydenoted a navigational device, a navigator or an optical mouse) and thespecially tinted reference surface (1513) on which it moves(alternatively referred to as a tinted or colored mouse pad). By theterm “3D”, is meant that the presently disclosed system, (1512) plus(1513), is able to support an electronic visual display (1507),typically controlled by a computer (e.g. 1514) on which a 3D object,cursor, or scenery can be maneuvered using a data set comprising threeor more parameters (e.g. x, y, z, θ, etc.). In practice, theseparameters can be used not only to produce translational motions, butalso rotational motions and, in some instances, to produce specialeffects. Note that for a high performance navigational system asdisclosed herein, all 3D motion vectors (e.g. translational, rotational,scaling, etc.) will preferentially be produced by a single gesturalmovement of the operator and, in fact, any cursor maneuvering that takesmore than one gestural movement to complete would be consideredredundant and non-ergonomic. The extraordinary capability of the presentsystem in providing a plurality of parameters by which to move a 3Dobject by one gestural movement of the operator is achieved by use of atinted reference surface (1513) and a color sensitive optoelectronicsensor (e.g. color CMOS image sensor or the like) which is embedded inthe presently disclosed navigation device (1512). We repeat, however,that the term “tinted” can apply to any treatment of the surface thataffects incident electromagnetic radiation by reflecting it, re-emittingit, changing its wavelength, phase or polarization in a manner thatpermits independent use of several non-interfering waves captured by theoptoelectronic sensor.

Referring again to FIG. 15C, the surface of the reference plane (1513)has, in this example, been “tinted” by some method of coloration (e.g.by red, blue, and green dyes/pigments for this case, or by alternativemethods with equivalent effects for other cases) to allow each point onthe reference plane to direct light of different wavelengths to anembedded image sensor. In fact, this can be done using methods otherthan reflection that have similar effects, such as interference,refraction, fluorescence or absorption and re-emission. On the referenceplane (1513), the variation (i.e., intensity, hue) of each colorationmaterial has a predefined profile (i.e., a surface pattern) that can berecognized by the image sensor embedded in the movable device (1612). Inan exemplary case, the reference plane is coated by the colorationmethod in three pure colors (e.g. red, blue, and green, etc.). Tofabricate this type of tinted planar reference surface, the deviceengineer may use a plastic sheet on which three spots are dispersedlypositioned (not shown in FIG. 15C), where the spots are doped with somegiven concentration of red, blue and green dyes/pigments that permitreflection at various intensities. The remaining points on the referencesurface can be doped using lower amounts of the dyes/pigments, but theyare no longer in one pure color as the three spots are, thedyes/pigments are thus in a blended mode in the remaining area. Thus,every geographic point on the tinted mouse pad will be colored by aunique combination of the three pure colors in accord with the varyingblending ratio of the three dying materials. It should be easilyunderstood that the other tinting methods may also be useful, but forthe simplicity of explanation, this embodiment uses red, green, and bluedyes/pigments.

The presently disclosed navigating device, in this exemplary form,comprises a set of light sources that are typically embedded within acavity that opens onto the surface (see FIG. 5). These sources emit aplurality of light beams in different colors (i.e. wavelengths) such asred, green, and blue. There are other ways of placing said light sources(e.g., embedding said light sources in the bulk material of referenceplane, like using a planar light guide within which light waves arepropagating via many reflections/refractions). An optoelectronic sensor,(e.g. a CMOS image sensor, CCD, etc.) also embedded in the device issensitive to reflections (or emissions) from the surface produced bythese light beams (e.g. RGB, CYMK, etc.). When the navigating devicemoves on the reference surface ((1513) in FIG. 15C), as a result of thevariations in light reflectance (or other forms of electromagneticemission as previously noted) from the reference plane, the photon fluximpinging on the optoelectronic sensor will be a mixed flux includingcontributions by various amounts (i.e. intensity) of photons inaccordance with a ratio established by the reflectance values of thespots being illuminated (i.e. in red, blue, or green colors). When thedevice moves on the mouse pad (1513), the optoelectronic sensor detectsthat the color index data representing each respective point has beenchanged (e.g. CIE 1931 RGB). As the navigator maneuvers all over themouse pad, a unique color gamut is formed based on the color index datadefined by the coloration (distribution of dyes) of the mouse pad(1513). The unique color gamut defined by the “tinted” reference surfacecan be of arbitrary shape or dimension, e.g. in FIG. 15A, it is shownschematically in rectangular shape (1516). In practice, the size of thisrectangular region (1516) can be smaller than that of the full colorgamut (1502). Nevertheless, as a consequence of the substantially highsensitivity of modern optoelectronic sensors, the size of the mouse pad(i.e., the area over which the color gamut is formed) will not cause anyrestrictions on the performance of the navigator. Theoretically, even ona small mouse pad of dimension one inch by one inch, the gamut canprovide millions of unique colors that will be resolved by the sensor.The abundant amount of color index data provided by the tinted mouse padcan satisfy the need for maneuvering a cursor on a large displayingdevice (e.g. a displaying panel 60 inch in diagonal).

When maneuvering a 3D object by both translational and rotationalmovement, measuring a color index from a tinted plane for motiondetection is a more robust way of tracking motion than doing patternrecognition on a desktop surface. This can be explained by reference toEq. (1). As Eq. (1) shows, when a navigator is intended to provide theposition of P′ for the computer, it has to know the whereabouts of pointP on the reference plane, i.e.,

$\begin{bmatrix}X \\Y \\Z\end{bmatrix}.$

Otherwise, the rotational matrix will always be multiplied by zero andthe P′ data will be reduced to a translation.

There is another requirement for the data

$\begin{bmatrix}X \\Y \\Z\end{bmatrix}\quad$

of Eq. (1), it must have high positional accuracy. This is because therange of X, Y, Z is usually far larger than that of translationaldisplacement data

$\begin{bmatrix}T_{x} \\T_{y} \\T_{z}\end{bmatrix}.$

On a mouse pad that is 10 in. square, the point resolvable by a typicaloptical mouse at 500 dpi resolution is 5000×5000. Thus, the address ofX, Y, and Z per pixel count falls in the range of about 0˜5000. Thebinary data representing the numerical data 5000 takes as many as 13bits. So, miscalculating X, Y, Z may easily introduce significant errorsto the Eq. (1). In the present system, it is the color, or hue, of thereflected light (i.e. optical flow in different colors) that providesthe data needed for motion detection. This is a method that is far moreaccurate than the block matching algorithm. As a consequence of thecapability of calculating the translational and rotational motion vectorresulting from a single gestural movement, maneuvering a cursor on a 3Dgraphic displaying device (1607) becomes relatively fast, closer torealism, and producing a positional resolution higher than that of theprior art. The following describes the calculation method used by thepresent system.

When the navigation device ((1512) of FIG. 15C) moves from point C to Don a tinted mouse pad (1513) under a constant illumination condition,the hue of the reflected light as seen by the cursor maneuvering device(1512) changes from point A to B in the color gamut (1502) regardless ofthe morphological condition of the mouse pad (1513). Since surfacemorphology (e.g. rough, smooth, glossy, etc.) does not affect the huedata drastically, each of the geographical positions on the mouse pad(1513) can be represented by a unique color index (e.g. CIE 1931 RGB,xyz, YUV, etc.). Specifically, every point in the color gamut (1502) canbe represented by three independent parameters (e.g. R, G, and B of thecolor index (e.g. CIE 1931)). These three parameters gain theirindependence from one another in that each optoelectronic receptor (i.e.pixel) is sensitive to only one of those colors (e.g. red pixel, bluepixel, and green pixel). Knowing this characteristic, the deviceengineer can tint the mouse pad (1513) with red, blue, and greendyes/pigments to create a specific gradient along each of thepre-defined directions (e.g. x-axis direction, y-axis direction, andz-axis direction) on the mouse pad (1513). The three independentparameters (e.g. R, G, B) on the 2D surface, can then be utilized as thereference data to designate the geographical points in a 3D graphicdisplaying device (1507).

Take FIGS. 15A and 15B as examples. FIG. 15B is a 3D coordinate system(1510) presented by a displaying device (1507), in which the originpoint is located at point P′; the three orthogonal axes (x-axis, y-axis,and z-axis) are located at {right arrow over (P′ Q′)}, {right arrow over(P′ R′)}, and {right arrow over (P′S′)}, respectively. FIG. 15A showsthe color gamut corresponding to the three dimensional coordinate system(1510), in which point P will be mapped to point P′, lines {right arrowover (PQ″)}, {right arrow over (PR″)}, and {right arrow over (PS″)} willbe mapped to lines {right arrow over (P′ Q′)}, {right arrow over (P′R′)}, and {right arrow over (P′S′)} in the 3D coordinate system (1510),respectively. When the disclosed navigator (1512) moves from point C toD (e.g. under constant illumination condition), the color index of thereflected light as is detected by the navigator (1512) moves from pointA (1508) to point B (1509) in the color gamut (1516). After dataconversion and transmission, a series of motion vectors (1515) orpositional data arrive at the computer or electronic system (1514), thecursor on the three dimensional displaying device (1507) moves frompoint A′ (1511) to B′ (1517) accordingly.

It is to be noted that both point A′ and B′ are located on the sameplane Q′R'S′. So, what the plane Q″R″S″ represents is not really a 3Dvolume; it represents a plane in 3D space. FIG. 15F schematicallydenotes the above situation more clearly. The reference planes (1519)and (1520) denote two planes in a 3D coordinate system. Mathematically,a plane (we call it Plane 7 hereafter) in a 3D coordinate system can beexpressed as the locus of a set of coordinates x, y, z, that satisfy:

$\begin{matrix}{{\frac{x}{A} + \frac{y}{B} + \frac{z}{C}} = 1} & (15)\end{matrix}$

Putting unit vector (n₁, n₂, n₃) into Eq. (15), the new equation can beexpressed as

$\begin{matrix}{{\left( {\frac{x}{n_{1}} + \frac{y}{n_{2}} + \frac{z}{n_{3}}} \right) \cdot \sqrt{A^{2} + B^{2} + C^{2}}} = 1} & (16)\end{matrix}$

Where the values of n₁, n₂, n₃ are determined by the followingequations.

$\begin{matrix}{n_{1} = {\frac{A}{\sqrt{{A^{2} + B^{2}} = C^{2}}} = \frac{A}{N}}} & (17) \\{n_{2} = {\frac{B}{\sqrt{A^{2} + B^{2} + C^{2}}} = \frac{B}{N}}} & (18) \\{n_{3} = {\frac{C}{\sqrt{A^{2} + B^{2} + C^{2}}} = \frac{C}{N}}} & (19)\end{matrix}$

Where n₁, n₂, and n₃ are the parameters of the unit vector normal toplane 7, and the parameter N (which equals √{square root over(A²+B²+C²)}) can be deemed as a “zooming” factor (i.e. scaling factor).When the value N (√{square root over (A²+B²+C²)}) is very large, itdenotes a situation that the object (i.e. the plane where the object ispositioned) is relatively close to the computer operator (we only takethe first quadrant into consideration), and vice versa. FIG. 15F showsseveral planes having the same unit vectors but different scaling factorN (1521, 1522, and 1523). When the navigator moves over the tintedreference plane, in cooperation with the scaling factor N, it can reachany point in the designated 3D coordinate system. FIG. 14B schematicallyshows a few exemplary designs (1408, 1409, 1410, etc.) of the presentlyinvented navigator associated with the input apparatus toincrease/decrease the scaling factor N (i.e., (1411) is a press button,(1412) is also a press button, (1417) is a touch-sensitive feature,(1413) is a wiggling knob . . . ). As FIG. 14B shows, the scaling factorcan be adjusted by the press button L (1411) or using (1413), awiggling/press button. Press button L (1411) is a zoom in/out device; itcombines two functions in one. When in default condition, the pressbutton L (1411) is disabled. To enable its functionality, press button Lmust be pressed down for a period of time. After then, depending on theposition of said press button L (or the force applied by operator'sfinger, etc.), the scaling factor N can be increased or decreasedaccordingly. Note that the act of pressing button L does not hinder thecalculation process of the navigator. Thus, while the navigator (1408)is maneuvering over the reference plane (1421), the operator can adjustthe scaling factor simultaneously. It is now understandable that thenavigator (1408) can reach any point in a 3D coordinate system.The press button (1411) can have many varieties in its generic design.For example, the design of (1417) literally is an integrated design ofthe touch sensing feature (1414) and zooming in/out button L (1411). Thedesign of wiggling/retracting knob (1413) has similar purpose as that of(1417), except said wiggling/retracting knob (1413) detects the wigglingmovement of the operator's finger by sensing its tilting angles (i.e.,the “wiggling” angle of said knob). There are other features that may goalong with said zooming/wiggling features (e.g., 1411, 1417, and 1413).For example, features (1412), (1415), (1418), (1419), and (1420) maydenote the conventional mouse buttons; they can collaborate with thesaid zooming in/out feature (e.g., 1411, 1417, or 1413) to denotecertain functionalities. Referring back to FIG. 15F, the plane (1521),(1522), and (1523) are three planes having the same unit vector butdifferent scaling factors (|{circumflex over (n)}|₁, |{circumflex over(n)}|₂, and |{circumflex over (n)}|₃). This can be implemented by thetouch sensing feature (1417). So, FIG. 15F can be compared to FIG. 14Dfor better understanding. As FIG. 14D shows, the entire surface of saidtouch sensing feature can be designated to three zones, i.e.,|{circumflex over (n)}|=const, denoting there is no zooming in/out;|{circumflex over (n)}|=|{circumflex over (n)}|−−, denoting the scalingfactor |{circumflex over (n)}| is decreasing; |{circumflex over(n)}|=|{circumflex over (n)}|++, which denotes the scaling factor|{circumflex over (n)}| is increasing. The operator must be noticed thatpoint (1524) denotes the point lying in a distance of infinity to theviewer; point (1524), namely, is the vanishing point to the graphicartists, as has been described in the background section. In FIG. 15F,the relative angle (e.g., ∠a₁O₁b₁, ∠a₂O₂b₂, etc.) formed by vanishingpoint of each plane (e.g., 1523, 1522, and 1521, etc.) and thecorresponding terminals (i.e., a₁, b₁, and a₂, b₂) will not be changedwhen the presently disclosed navigator maneuvers on the reference plane(1521), (1522), or (1523). This unique feature (perspective sketching)provides an extraordinary convenience for the people in electronicgraphic art profession. That is, the graphic artists can zoom in and outthe scene to perform his/her professional work on different layers(e.g., 1521, 1522, 1523) without worrying changing the perspective angleinadvertently. The vanishing points O₁, O₂, and O₃ will always reside onthe same geographic point of reference plane (1521), (1522), (1523),etc. So, the graphic artist can align his/her hand movement on thereference plane to the scene he sees in the displaying device, much likehow a sketching artist normally do in the studio (i.e., the above statedalignment is a plane-to-plane one).

There are many occasions where a graphic artist would want to draw acurve, produce a genre by strokes, or use finesse to improve the work.This has to do with the non-linear motion of the artist's hand, wherelinear motion implies a hand motion only generates translationalmotions.

$\begin{matrix}{{\frac{T_{x}}{n_{1}} + \frac{T_{y}}{n_{2}} + \frac{T_{z}}{n_{3}}} = 1} & (20)\end{matrix}$

It must be acknowledged that all the translational motion vectorsidentified by planes (1521), (1522) and (1523) of FIG. 15F satisfy Eq.20 (their unit vectors are the same). To reach a point (or a pixeladdress) that may be located out of plane (1521), (1522) and (1523)etc., the disclosed navigator has to resort to the rotational matrix Rof Eq. (1). The associated methods have been explained in the previoussection. Thus, by a rotational movement which may take placeconcurrently with the translational movement, the address of point P′ asis calculated by Eq. (1), and the new position P′ may fall out of plane(1521), (1522) and (1523). Compared to the motion vector derived by Eq.(20), one may conclude that the scenery controlled by Eq. (1) is a 3Dobject, whereas that of Eq. (20) is 2.5 D. Naturally, the 3D one givespeople a feeling that the object is live, but the vanishing point is nolonger being positioned on a fixed point of the reference plane (tocertain operators, this kind of scene may be exceedingly dynamic). Whatis a big blessing is that the operator can switch between said two modes(2.5D or 3D mode) relatively easily; and because of the hue detectingcapability of the presently disclosed navigator, the positional accuracyin either mode is maintained at unprecedented high resolution andsensitivity.

At this stage, the presently disclosed navigator (1512) is apoint-to-point system; for every geographical point on the tintedreference surface (1513), there lies one and only one point on the 3Ddisplaying device whose address corresponds to the whereabouts of suchgeographical point. On the other hand, the present invention reservesthe capability for relative motion detection. When working in relativemotion detection mode, the data stream sent by said cursor maneuveringdevice (1612) can be formed in delta format (i.e. (Δx₁, Δy₁, Δz₁) and(Δx₂, Δy₂, Δz₂)). In practice, the operator may choose either mode thathe/she desires. When the presently disclosed navigator (1512) works inthe relative motion sensing mode, it will continually measure thedifference of the intensities of the red, green, and blue light alongwith the relative movement between said three dimensional cursormaneuvering device (1512) and mouse pad (1513). After the conversion byEq. (6), the data sent to the remote computer is a positionaldisplacement data (i.e. (Δx, Δy, Δz)) for cursor maneuvering use.

The presently disclosed navigator may collaborate with the conventionalart to generate some unique functionality. For example, when bothmethods are used (i.e. pattern recognition and color detection), therewill be two independent motion vectors generated by a single gesturalmovement of the operator's hand instead of just one. The first motionvector is acquired by the “tinting” condition of the mouse pad, which isa property of the present disclosure. The second motion vector has to dowith pattern recognition, which is taught in NU11-007. When used forderiving the rotational motion vector, the data acquired by the firstmethod (acquired by tint detection) and the second method (acquired bypattern recognition) can be compared to yield a rotational motion vectorω. Thus, just by a single gestural movement of the presently disclosedcursor maneuvering device (1512), a 3D motion vector (Δx, Δy, Δz) plus arotational motion vector (ω) can be derived simultaneously. The new setof motion vectors derived by the single gestural movement of theoperator's hand becomes a four parameter one vector: (Δx, Δy, Δz, ω).This result profoundly empowers the next generation three dimensionaldisplay industry.

The present disclosure offers a unique flexibility of working indifferent environments. When placed on a desktop surface without thetinted mouse pad (1513), the presently disclosed cursor maneuveringdevice (1512) works as a 2D optical mouse. When placed on the 3D tintedmouse pad (1513), the presently disclosed cursor maneuvering device(1512) works as a three dimensional cursor maneuvering device. In eithercase, the presently disclosed cursor maneuvering device (1512) allowsthe operator to obtain six degrees of cursor movement while onlyphysically moving his or her hand on a 2D plane (e.g. the desktopsurface or mouse pad); the six degrees of cursor movement (i.e. X, −X,Y, −Y, Z, and −Z) is in fact the result of the generic property of light(i.e. what we learn from the physics of electromagnetic radiation).There is no need to roll a ball or wiggle a joystick as the conventionalarts have to do. The presently disclosed device offers an unprecedentedadvantage for the next generation cursor maneuvering device. It is ahigh resolution and high sensitivity three dimensional cursormaneuvering device in the shape of a conventional optical mouse, and itis able to control the movement of objects or cursors in a threedimensional displaying device (e.g. 3D flat panel display) withoutphysically departing from the 2D desktop surface.

It is also important to illustrate the capability of the presentlydisclosed device in terms of microelectronic circuitry design. FIGS.15A, B, and C schematically illustrate a way of implementing thepresently disclosed navigator (1512) without using a sophisticated DSP(digital signal processor), equivalent to the microprocessor in the ICindustry. The 3D motion detection circuitry in the cursor maneuveringdevice is designed to measure the tint of mouse pad (1513), it onlyneeds to sense the variations in hue (i.e. no pattern recognition isrequired). This makes the disclosed device simple, highly sensitive,with high resolution, high speed, economic in power consumption, andergonomic. In an extreme case, the DSP may be totally removed, but thisis not a prerequisite of the present invention.

Embodiment 1

Embodiment 1 teaches a method of forming a planar reference surface(e.g. mouse pad (1513) of FIG. 15C) which can be used by the presentsystem to create a 2.5D cursor maneuvering device (1512). Embodiment 2will extend the performance capabilities of the device to that of a 3Done.

As FIGS. 20A, 20C, and 20E shows, a tinted reference plane (1513) can befabricated by screen-printing means, or whatever means of similareffect. In FIGS. 20A, 20C, and 20E, each figure has a specific profile(light intensity in the respective wavelength regime) along one axis(e.g., x, y, or z axis or another axis specified by the system). Thus,the apparent color of each of the respective points on the surface is acombination of, for example, red, blue, and green (In FIGS. 20B, D, andF, they are represented generally by the color index intensities I_(x),I_(y), and I_(z)). For example, as FIG. 20A shows, along the horizontalaxis, from 0 to x, the color index I_(x) has been chosen to be alinearly decreasing one; this phenomenon is depicted by FIG. 20B. Usingthe same method, one may designate the profiles for the other colorindex, i.e., along the vertical axis (i.e., y axis). The third colorindex I_(z), the one that denotes the z axis, varies its respectivevalue along the diagonal direction (e.g., 45° to the x axis).

We take point P for analysis; we arbitrarily pick a point at about thegeometrical center of the reference plane (1513); point P hence producesa mixed colored light (a mixture of radiation wavelengths) comprisingabout half of the maximum of I_(x), half of the maximum of I_(y), andhalf of the maximum of I_(z). For all points that reside on the right ofpoint P, the respective color index value I_(x)′ is higher than of themaximum of I_(x), and vice versa. Similar rules (of coloration) mayapply to the other two color indexes, i.e., I_(y) and I_(z). Thus,namely every point on a reference plane conjunctly formed by FIGS. 20A,C, and E has a unique combination of the color index (I_(x), I_(y), I₁).These color indexes (I_(x), I_(y), I_(z)) are in fact the parameters (A,B, C) referred to in Eq. (6). Through Eq. (6) one is able to derive thegeographical address data (X, Y, Z). We then use such geographical data(X, Y, Z) to gain access to the respective points in 3D coordinatesystem. Thus, we have mapped a 2D plane into a 2.5D space.

We must point out that the coordinate system formed by FIGS. 20A, C, andE is a 2.5D instead of a fully 3D one since the angle formed by the xaxis and z axis is a fixed one (e.g., 45°). To make said coordinatesystem a genuine 3D one, such an angle must be adjustable, and thisshall be elaborated in embodiment 2.

The point P has a special usage, it can serve as the vanishing point ina perspective sketching graphic rendering system.

A 2.5D cursor maneuvering system (i.e., cursor maneuvering device (1512)plus a 2.5D reference plane) can collaborate with the 2.5D displayingdevice seamlessly. We refer now to schematic FIGS. 21 A and B. A globe(2102A) is presented by a 2.5D displaying device (2101A); throughcollaboration between the presently disclosed 2.5D cursor maneuveringsystem ((1512) and the 2.5D reference plane (2101B)), the operator isable to access to all points on the globe that he/she can see from aspecific perspective angle (e.g., 45°). Those points that cannot be seenby the operators at one time (e.g., points on the back of the globe,such as point N) would require an adjustment of z value whenever theoperator desires to access to them.

In embodiment 1, the range of depth value z denotes the depth of scene.Thus, by moving the cursor maneuvering device (1512, not shown in FIG.21B) from point M′ to N′ on the reference plane (2101B), the depth valuez as is detected by the presently disclosed navigator changesaccordingly. The cursor on the displaying device (2101A) henceforth canbe moved from M to N (N is invisible to the operator at this moment). Ashas been explained in former paragraphs, the 3D geographic mappingrelationship between the displaying device (2101A) and reference plane(2101B) facilitates the operator to build up his/her finesse in 3Dapplications, which is extremely important to the careers that requirethe dexterity.

If the depth value provided by said 2.5D cursor maneuvering system isnot sufficient to cover that of the 2.5D displaying device (2101A), onemay resort to Eqs. (2) and (6) to numerically expand the depth value.That is, based on same set of color index (A, B, C), elementsF_(A=1, B=0, C=0), F_(A=0, B=1, C=0), or F_(A=0, B=0, C=1) shall beincreased to yield larger Z values (this can be done automatically sinceEq. (6) and the respective elements are stored in presently inventedcursor maneuvering device 1512).

Embodiment 2

Embodiment 2 is an extension of embodiment 1, from a 2.5D cursormaneuvering system to a genuine 3D one.

As has also been defined in the background, the term 2.5D means that thetechnique used has the depth value, which the 2D one does not, but itcannot provide the full rotational function; the 3D one does.

It is now clear that human hand usually has linear and non-linearmotions, conventional optical mouse has tossed away the non-linearportion of mouse movement. For a reference plane formed by a constanttinting mean (e.g., dying, pigmenting, etc.), the motion data acquiredby hue is a 2.5D type (i.e., there are x, y, and z data, but theperspective angle is fixed). Note that from the said tinted referenceplane, a second motion data can be acquired by the pattern recognitionfeature embedded in the presently disclosed cursor maneuvering device.This pattern recognition feature is nothing different than those of theconventional optical mouse. Contrary to said first set of motion data(acquired by hue detection); said second set of data has more to do withthe surface morphology (e.g., dust particles that can be targeted bysaid pattern recognition feature), and its result shall not be affectedby hue variations by and largely. For example, if one of the three lightsources (e.g., 503, 505, or 507 of FIG. 5) of the presently disclosedcursor maneuvering device slightly changes its illumination condition,then the motion vector data measured by said pattern recognition featurewill not be affected much, but the one associated with tint measurementfeature will. In essence, when the presently disclosed cursormaneuvering device moves on the reference plane (1608B), the tint valuemeasuring feature provides a first data set (X_(i), Y_(i), Z_(i)), thepattern recognition feature provides a second of data set, (X_(ii),Y_(ii), Z_(ii)), respectively. In practice, the second feature may useEq. (5A) and (5B) to convert a set of (x, y) data into (x, y, z), thedifference between the two motion data can be expressed as the following(Eq. (21)).

(ΔX,ΔY,ΔZ)=(X _(i) ,Y _(i) ,Z _(i))−(X _(ii) ,Y _(ii) ,Z _(ii))  (21)

Thus, Eq. (21) provides a 3D motion vector (i.e., ΔX, ΔY, ΔZ) in anunprecedented way; it denotes the non-linear portion of the movement ofthe presently disclosed cursor maneuvering device. (ΔX, ΔY, ΔZ) can beused together with the translational motion vector to control certainnon-linear movement of the object in the displaying device. Endowed withsuch a unique feature, a 2.5D displaying device now can be characterizedas a 3D one.

For example, if an operator desires to set the magnitude of a rotationalmotion vector significantly larger than the presently disclosed cursormaneuvering devices originally has provided, one way to do it is tochange the tinting condition of the reference plane by optical means(e.g., 510). This can be explained by FIG. 5. If judged from outlook,the presently disclosed cursor maneuvering device (i.e., embodiment 2)may look like a conventional optical mouse. However, the cursormaneuvering device (502) shown in FIG. 5 differs from the conventionalart in that said cursor maneuvering device is affiliated with a colorsensitive optoelectronic sensor (509) and three light sources (i.e.,503, 505, and 507), each of which emits a light ray in different colors(e.g., red, blue, and green; other color are also permissible); asimilar device is shown in FIG. 15 (1512), which is a simplified modelof (502), allowing multiple light rays to impinge on the reference plane(1513) from one or more lighting fixture (the origin of light source forreference (1513) may vary; they are not limited to the situation asdesignated by item 503, 505, or 507 of FIG. 5).

Note that in both FIGS. 15 and 5, the respective optoelectronic sensors(e.g., 509) are color sensitive ones. Thus, like FIG. 15A, said cursormaneuvering device can form a color gamut of its own (i.e., whose shapeis presented by (1502). Note that the positions of point Q, R, and S mayshift due to the variation of the respective light intensities. So wewill use FIGS. 15A and C to illustrate what happens when one lightsource (e.g., 503, 505, or 507 in FIG. 5) changes its output power (orcertain characteristic, such as polarity, shutter speed of saidoptoelectronic sensor, etc.). It is also to be noted that the outputpower of the three light sources (503, 505, and 507) are independentlycontrolled by the driving circuitry of the cursor maneuvering device.Thus, when the intensity of one light source changes, the output powerof the other two light sources can be maintained unchanged. Under thiscircumstance, the point of equal intensity (e.g., point P in FIG. 15A)will shift its position toward or away from the light source thatchanges its output power. Consequently, the origin point P′ of thecoordinate system in the 3D displaying device (1507) may change itsposition along one axis (e.g., {right arrow over (P′Q′)}, {right arrowover (P′R′)}, or {right arrow over (P′S′)}) corresponding to the outputpower adjustment of one lighting source (e.g., 503, 505, or 507). Thus,when the presently disclosed invention moves though the samegeographical path on the reference plane (510), a new set of colorindices (e.g., A′, B′, C′) is acquired. This new set of color index inturn, through the conversion process provided by Eq. (6), provides a newset of motion/position data, i.e., (X_(i)′, Y_(i)′, Z_(i)′). FIG. 15Dshows that a microscopic feature (e.g., 1518B) on the reference plane(1507D) will not change its morphology due to various light tintingconditions. If an operator moves the presently disclosed cursormaneuvering device on said tinted reference surface, themorphology/microstructure of said reference plane (510) will not bechanged, henceforth the second set of motion vector/positional data(usually measured by BMA method, monochromatic images) is not muchchanged, it is still (X_(ii), Y_(ii), Z_(ii)). Putting the new first setof data and old second set of motion vector data into Eq. (21), onederives Eq. (22) which is the following:

(ΔX′,ΔY′,ΔZ′)=(X _(i) ′,Y _(i) ′,Z _(i)′)−(X _(ii) ,Y _(ii) ,Z_(ii))  (22)

One may find that the rotational motion vector has changed (i.e., from(ΔX, ΔY, ΔZ) to (ΔX′, ΔY′, ΔZ)). This is equivalent to say that therotational motion vector can be manipulated by adjusting the tinting(i.e., lighting) condition of the reference plane (510).

We will take an imagined Disney® Video Program as the example of usingthe above stated four parameter motion vector. We visualize a swordfightbetween Peter Pan and Captain Hook. As compared to a previous 2D systemin which an operator cannot rotate Peter to face Captain Hook as heflits around Captain Hook, this time the rotational motion vector ofPeter is changed by the computer program in a timely fashion (i.e., theoutput power of the respective light sources is timely adjusted inaccordance with the status of computer program). When Peter Pan nowflies around Captain Hook through a succession of three dimensionalarcs, his body gestures keep changing. That is, when Peter Pan is onCaptain Hook's right, his face will turn left to face Captain Hook. WhenPeter Pan flies to Captain Hook's left, his face will turn right.Through the help of Eq. (21) and different tinting condition on thereference plane (510), the operator is able to derive the desired motionvectors for Peter Pan. To reiterate, the relative weigh betweentranslational motion and rotational motion (i.e., the non-linearportion) of an object's movement can be adjusted by the presentlydisclosed cursor maneuvering device.

Embodiment 3

In embodiment 3, the cursor maneuvering device has fundamentally changedits hardware structure from that of a desktop mouse to a 3D touch pad.From its appearance, the entire hardware of the device resembles thatdisclosed in NU11-002. The major difference between the device of thisembodiment and the teachings of NU11-002 is that the present deviceconverts the tri-stimuli color index data (e.g. CIE 1931 RGB) into a 3Dcoordinate one (X, Y, Z). The device provided by NU11-002 technique, onthe other hand, knows nothing about this optical physics—it onlyprovides 2D data (X, Y).

Referring now to schematic FIG. 22, the navigational device of thisembodiment is formed from a cavity (2204) which is covered with a lightmixing plate (2203), e.g. a Lambertian plate, a light diffusing plate,or a fluorescent plate. The cavity (2204) contains a plurality ofcolored light sources (i.e. 2205, 2207, 2210), which are constructed of,for example, LEDs (light emitting diodes), where each light source emitslight in a color (i.e. wavelength) different than that of the other. Thecavity (2204) also contains a photo sensor (2206), and the cavity (2204)can be formed into different shapes, e.g. square, rectangular, circular,oval, etc. The plurality of light sources are positioned in a manner onthe base of the cavity (2204) as separated as possible, and the photosensor (2206) is positioned on the base of the cavity (2204) amongst theplurality of colored lights in such a way as to allow the image sensorto detect the image of the pointing device in contact with the surfaceof the light mixing plate.

The light mixing apparatus (e.g. a Lambertian plate, a diffusing plate,or a fluorescent plate, etc.) plate (2203) is a semi-transparent platethat has a surface, which is a nearly perfect diffusing surface. Thelight mixing plate is semi-transparent, which allows light from eitherside of the plate to pass through in a highly diffused state, but theobserver is not able to discern the exact shape of any object that liesat a distance from the opposite side of the plate. The light mixingplate (2203) that covers the cavity (2204) disperses the ambient lightfalling on its surface, which is external to the cursor maneuveringdevice, and the light mixing plate (1403) does the same work for thelight beam coming from within the cavity when it passes through thelight mixing plate. As of such, a blurred image without clearly definedcontours will be formed on the photo sensor (e.g. 2206). Mathematically,the light mixing effect can be modeled by radiative transfer equations(RTE). For example, a rough surface would scatter light in moredirections (i.e. more isotropic) than a smooth surface does. Thus theclarity of the image caused by the light beam being reflected by the twodifferent kinds of surfaces differs as a result of different scatteringeffects.

The image blurring phenomenon is enhanced when an object is moved to adistance farther away from the light mixing plate (2203), and the imageblurring is reduced as the object is moved closer to the light mixingplate (2203). There are other methods and types of plates that may servethe same purpose of passing a mixed light to the image sensor. Forexample, a transparent plate with a pointing device being able to mixthe incoming light and reflect it back to the image sensor will allowthe image sensor to track the pointing device based on the colorreflected back to the image sensor. In the conventional art, an objectwith the blurred edge poses a challenge for pattern recognition, andmakes the task of motion detection nearly impossible.

However, in the present disclosure, the task of motion detection is notrendered impossible since it is the color of the object, or pointingdevice, rather than the edges, or contours, of the object thatdetermines the location. The light intensity value of the object, orpointing device, being tracked has the highest value when the object, orpointing device, is touching the mixing plate, the blurred image of theobject, or pointing device, has a higher light intensity than that ofthe surrounding area of the mixing plate. This bright but blurred spotserves as an ideal target for the image sensor to trace when the object,or pointing device, is moving on the light mixing plate. The pluralityof light sources within the cavity are preferably LED devices; however,other light sources can be used. The image sensor is preferably acolor-sensitive CMOS imaging device capable of viewing the insidesurface of the light mixing plate covering the cavity; however, othercolor imaging devices capable of viewing the color of the image of theobject, or pointing device, rendered from the light mixing plate can beused to determine the location of the object, or pointing device.

When a pointing device (2202A, B, or C), e.g. a finger, stylus orsimilar device, contacts the surface of the light mixing plate, aportion of light emitted from the plurality of light sources within thecavity of the cursor-maneuvering device produces an brightened image ofthe of the pointing device (i.e. by reflecting light back to thecavity). The photo sensor (2206) then detects the color hue of thisbrightened image and monitors the changes of the hue as the pointingdevice is moved over the surface of the light mixing plate from whichthe electronic circuitries associated with the image sensor analyzes thecolor hue and provides control to a cursor on the screen of an portableelectronic device, e.g. notebook PC, cellular phone, global positioningsystem, gaming device, etc. Further, the pointing device can be liftedand reapplied to the light mixing plate's surface to indicate selectionof an element on the display of the electronic device. Still further andespecially in the case of the notebook PC, multiple fingers (e.g. 2202A,B, and C) can be used in various predetermined locations (relative orabsolute) on the light mixing plate's surface to perform functions inaddition to cursor movement, such as scrolling, rotating an image orzooming in or out, on the display screen to provide high resolution andsensitivity for accurate positioning of the cursor.

TABLE 1 is an example of a lookup table for the RGB format and thetranslation of a particular set of colors of the contact area as seen bythe image sensor translated into the location of the pointing device onthe light mixing plate.

TABLE 1 Corresponding geographical Color index value position on ofimage sensor light mixing plate R1 G1 B1 X1 Y1 Z1 R2 G2 B2 X2 Y2 Z2 R3G3 B3 X3 Y3 Z3

The color index used in Table 1 is not the only usable one. A uniquecolor code can also be created for the other color formats, for exampleYUV, LAB, CIE, etc. (these are color index values). Thus, by detectingthe color of light reflected from a pointing device the image sensor andassociated electronic circuits can determine the 3D position andmovement of the pointing device and translate the location and movementof the pointing device into 3D location and movement of the cursor on adisplay associated with the electronic device to which the cursormaneuvering device is connected. Mathematically, one also can derive the(X, Y, Z) data through Eq. (6), in which case there is no need, or areduced need, for the lookup table (a lookup table would require certainamount of memory).

In essence, it is to be noted that the presently disclosed device (i.e.embodiment 3) enjoys the following advantages.

-   -   (1) The reference plane (i.e. Lambertian plate 2203) is a 2D        structure endowed with touch sensing capability; there is no        need for a pointing device to move in a 3D space in order to        acquire a 3D positional data.    -   (2) The reference plane (i.e. Lambertian plate 2203) is        stationary—it suppresses the error caused by human body        vibration (which is abundant in magnetic trackers). (3) The 3D        coordinate system constructed by the three-light-source feature        of the present disclosure can be mapped to that of the 3D        rendering device directly (e.g. point-to-point), making the        presently disclosed cursor maneuvering device (3D touch pad)        suitable to perspective sketching works.    -   (4) The presently disclosed invention has very high sensitivity        and high resolution, this is largely attributed to the image        sensor used (e.g. CMOS image sensor)    -   (5) Zooming in and out, changing vanishing point, or rotation,        like those operations described in embodiment 2, can be done        easily. That is, the scale of the coordinate system can be        adjusted in a timely fashion either by lighting condition,        amplifier gain setup for AD conversion process, or numerical        calculation (as Eq (6) does).

Embodiment 4

Embodiment 4 discloses a unique reference surface (e.g., a tinted mousepad) whose tinting materials are not necessarily in red, blue, or greencolors. In fact there are many substances capable of reflecting,refracting, interfering with or photo-emitting electromagnetic waves ina specific wavelength that may not be in the visible range of red, blue,or green. Together they can also form a unique color gamut (e.g., YMC;Yellow, Magenta, and Cyan). There are even extreme cases in which thereference surface reflects or emits electromagnetic waves that are notnecessarily in the visible light regime; or, where the electromagneticwaves carry certain identifiable characteristics other than wavelength(e.g., polarization, phase) that can serve instead of wavelength touniquely characterize absolute position on a reference surface. Innature there are many substances that emit, refract or reflect visible,UV or IR radiation when they are irradiated by corresponding waves.Using the same methods taught by the present disclosure, three lightreflecting/refracting ingredients can also form a unique “color” gamut,so that they also can be used to form a reference surface for thepresently disclosed 3D navigator to use. Their effect will be equivalentto what was taught by the present disclosure when applied to reflectedvisible radiation: namely all of the geographical points on saidreference surface can provide tri-stimuli indexes that are differentfrom one another and can be combined independently to form a mappingfrom a 2D surface to a 3D space. From these tri-stimuli index data, onecan derive the positional data (X, Y, Z), which is similar to what (Eq.6) does.

The coloration profile of a reference plane as depicted as above can besomewhat like those of FIGS. 20A through 20E, but different profiles arealso allowed (e.g., CIE 1931 YMC, etc.). The fundamental physics is thesame—the intensity of radiation reflected from or otherwise emitted bythe reference plane by any of several methods including refraction,photo-emission, interference, etc., can be denoted as I′_(x), I′_(y),and I′_(z). What makes the intensities I′_(x), I′_(y), and I′_(z)different from those of FIGS. 20A, C, and E is(are) that they may changevalues along the directions that are necessarily the x, y, or z axes inFIG. 20A˜E. In additions, the I′_(x), I′_(y), or I′_(z) in embodiment 4do not necessarily denote RGB or even any visible light. For example,the device engineer can designate the light source used by embodiment 4to be UV, IR, etc. In practice, UV light can provide better positionalaccuracy (i.e., wavelength is shorter), IR sensor can provide betterimmunity to the noise from room ambient light.

As is understood by a person skilled in the art, the preferredembodiments of the present disclosure are illustrative of the presentdisclosure rather than being limiting of the present disclosure.Revisions and modifications may be made to methods, processes,materials, structures, and dimensions through which a 3D maneuveringsystem utilizing both color detection and, optionally, pattern detectionon a tinted or non-tinted planar reference surface is made and used,while still providing such methods, processes, materials, structures anddimensions in accord with the present disclosure as defined by theappended claims.

What is claimed is:
 1. A maneuvering system configured for: 1)maneuvering a 3D object in a 3D graphical rendering system and: 2)controlling 3D motion in an electronic or electromechanical device,wherein 1) and 2) are simultaneously achieved by using two independentsets of data obtained from a single planar reference surface illuminatedby said maneuvering system wherein said two independent sets of data areobtained from the analysis of electromagnetic radiation of variablewavelengths and intensities emitted by said illuminated planar referencesurface, and from motion detection of targeted topological features onsaid planar reference surface.
 2. The maneuvering system of claim 1wherein analog circuitry is used for analysis of electromagneticradiation from said planar reference surface and wherein digitalcircuitry is used for analysis of the relative motion of targetedtopological features on said planar reference surface.
 3. Themaneuvering system of claim 1 wherein said analog circuitry comparesmultiple sets of hue intensity data from said electromagnetic radiationof variable wavelengths and wherein said analog comparison provides adirection of motion or a displacement value for a cursor or graphicallyrendered object represented in a three-dimensional coordinate system. 4.A method of motion detection comprising: providing a device configuredfor illuminating a targeted topological feature on a two-dimensionalsurface by the emission of electromagnetic radiation characterized bywavelengths, intensities, polarizations and phases that areindependently variable, then; by measuring both the relativedisplacement of said targeted topological feature and variationsproduced in said wavelengths, intensities, polarizations and phase ofsaid illuminating electromagnetic radiation resulting from theirreflections from said targeted topological feature, determining a linearmotion and a non-linear motion of said targeted topological feature. 5.The method of claim 4 wherein said device simultaneously provides twoindependent sets of data obtained from a single planar reference surfaceilluminated by said device wherein said two independent sets of data areobtained from the analysis of electromagnetic radiation emitted by saiddevice wherein said electromagnetic radiation is of independentlycontrolled variable wavelengths and intensities striking said planarreference surface and thereupon being emitted by or reflected by saidilluminated planar reference surface, and, independently, from motiondetection of targeted topological features on said planar referencesurface.
 6. The method of claim 4 wherein said device includes analogcircuitry that is used for analysis of electromagnetic radiation emittedby or reflected from said planar reference surface and wherein digitalcircuitry is used for analysis of the relative motion of targetedtopological features on said planar reference surface.
 7. The method ofclaim 4 wherein said targeted topological feature may be a temporaryfeature such as a dust particle or it may be a permanent morphologicalfeature inherent in said surface.